AWS Lambda auto scaling improvements for Amazon MSK and self-managed Apache Kafka as event sources
AWS Lambda has launched improved auto scaling for Amazon MSK and self-managed Kafka as event sources to improve performance and help lower costs for customers. Lambda starts with one consumer and checks the OffsetLag metric (measure of backlog at source) every minute and scales up or down every 3 minutes. Lambda allows up to one consumer per partition per topic for a Kafka cluster. Previously, Lambda used to start connection with Kafka clusters at 50% of maximum number of consumers, check for OffsetLag, and scale up consumers every 15 minutes.
AWS Lambda has launched improved auto scaling for Amazon MSK and self-managed Kafka as event sources to improve performance and help lower costs for customers. Lambda starts with one consumer and checks the OffsetLag metric (measure of backlog at source) every minute and scales up or down every 3 minutes. Lambda allows up to one consumer per partition per topic for a Kafka cluster. Previously, Lambda used to start connection with Kafka clusters at 50% of maximum number of consumers, check for OffsetLag, and scale up consumers every 15 minutes.
For more details, please see Auto scaling of the Amazon MSK event source and Auto scaling of the Kafka event source. This feature is available in all regions that AWS Lambda is available in.