Amazon EMR Serverless now supports large worker sizes to run more compute or memory intensive workloads
Amazon EMR Serverless is a serverless option in Amazon EMR that makes it simple for data engineers and data scientists to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers. An EMR Serverless application internally uses workers to execute your workloads and you can configure different worker configurations based on the need of your workload. Previously, the largest worker configuration available on EMR Serverless was 4 vCPUs with up to 30 GB memory. Today, we are excited to announce that EMR Serverless now offers worker configurations of 8 vCPUs with up to 60 GB memory and 16 vCPUs with up to 120 GB memory, allowing you to run more compute or memory-intensive workloads on EMR Serverless.
Amazon EMR Serverless is a serverless option in Amazon EMR that makes it simple for data engineers and data scientists to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers. An EMR Serverless application internally uses workers to execute your workloads and you can configure different worker configurations based on the need of your workload. Previously, the largest worker configuration available on EMR Serverless was 4 vCPUs with up to 30 GB memory. Today, we are excited to announce that EMR Serverless now offers worker configurations of 8 vCPUs with up to 60 GB memory and 16 vCPUs with up to 120 GB memory, allowing you to run more compute or memory-intensive workloads on EMR Serverless.
Larger workers can help you improve runtime performance of your jobs. If your job is shuffle heavy, using larger workers can reduce inefficient data transfers between executors. If your job suffers from data skew, larger workers reduces the chances of out of memory failures. Additionally, if your job needs to cache data, larger workers allow you to cache more data, boosting job performance. To take advantage of these benefits, we recommend using larger workers in EMR Serverless for your compute and memory-intensive Spark and Hive workloads.
To learn more about different worker configurations, please visit our documentation. Large workers are available in all AWS Regions where EMR Serverless is available.