AWS Glue now supports additional configuration options for memory-intensive jobs
You can now specify a worker type for Apache Spark jobs in AWS Glue for memory intensive workloads.
You can now specify a worker type for Apache Spark jobs in AWS Glue for memory intensive workloads.
Previously, all Apache Spark jobs in AWS Glue ran with a standard configuration of 1 Data Processing Unit (DPU) per worker node and 2 Apache Spark executors per node. You can now pick from two new configurations, G.1X and G.2X, that provide more memory per executor.
To learn more about these configuration options, please visit our documentation. These options are now available in all the AWS regions where AWS Glue is available except the AWS GovCloud (US-West) Region.