AWS Glue now supports additional configuration options for memory-intensive jobs submitted through development endpoints
You can now specify additional worker types when you use AWS Glue development endpoints. An AWS Glue development endpoint is a serverless Apache Spark environment that you can use to develop, debug, and test your AWS Glue ETL scripts in an interactive manner.
You can now specify additional worker types when you use AWS Glue development endpoints. An AWS Glue development endpoint is a serverless Apache Spark environment that you can use to develop, debug, and test your AWS Glue ETL scripts in an interactive manner.
Previously, you were able to specify the additional worker types only for Apache Spark jobs in AWS Glue. You can now pick from two new configurations, G.1X and G.2X, that provide more memory per executor.
To learn more about these configuration options, please visit our documentation. These options are now available in all the AWS regions where AWS Glue is available except AWS GovCloud (US-East) and AWS GovCloud (US-West).