AWS Step Functions launches large-scale parallel workflows for data processing and serverless applications

AWS Step Functions expands support for iterating and processing large sets of data such as images, logs and financial data in Amazon Simple Storage Service (Amazon S3), a cloud object storage service.

AWS Step Functions is a visual workflow service capable of orchestrating over 10,000 API actions from over 220 AWS services to automate business processes and data processing workloads.

Now, AWS Step Functions can iterate over objects such as images or logs stored in Amazon S3, then launch and coordinate thousands of parallel workflows to process the data, and save the results of executions to Amazon S3. You can scale thousands of parallel workflow executions, running concurrently, with the new distributed mode of the Map state. You can use the distributed Map mode to analyze millions of log files for security risks, iterate terabytes of data for business insights or scan images, and video files for specific objects. To process your data, use compute services such as AWS Lambda and write code in any language supported, or choose from over 220 purpose-built AWS services to accelerate your development. AWS Step Functions manages the concurrency and error thresholds, provides visibility into the progress of each individual workflow execution in a visual operator console, and gives you control over how faults and errors are handled.

Start using the new distributed Map mode today in the console with Workflow Studio, Command Line Interface (CLI) or SDK. To learn more, please see the AWS Step Functions developer guide and the launch blog.

AWS Step Functions’ distributed Map will be generally available in the following regions: US East (Ohio and N. Virginia), US West (Oregon), Canada (Central), Asia Pacific (Tokyo, Singapore and Sydney) and Europe (Ireland, Frankfurt and Stockholm). For a complete list of regions and service offerings, see AWS Regions.