AWS Glue now provides the Apache Spark UI to monitor and troubleshoot Glue ETL jobs
Starting today, you can use the Apache Spark UI to monitor and inspect Glue ETL jobs. The Apache Spark UI provides a web interface to visualize a Directed Acyclic Graph (DAG) representing different stages in Glue ETL job execution, analyse Spark SQL query plans and check the event timeline of running or terminated Spark executors. The Spark UI can be used in conjunction with job metrics and continuous logging to monitor and troubleshoot complex Glue jobs. The Spark UI is available for both Glue ETL jobs and Glue Development Endpoints.
Starting today, you can use the Apache Spark UI to monitor and inspect Glue ETL jobs. The Apache Spark UI provides a web interface to visualize a Directed Acyclic Graph (DAG) representing different stages in Glue ETL job execution, analyse Spark SQL query plans and check the event timeline of running or terminated Spark executors. The Spark UI can be used in conjunction with job metrics and continuous logging to monitor and troubleshoot complex Glue jobs. The Spark UI is available for both Glue ETL jobs and Glue Development Endpoints.
This feature is available in all regions where AWS Glue is available.
To learn more about this feature, please visit our documentation.