The new distributed map state can launch up to ten thousand parallel workflows to process data. csv files stored in Amazon Simple Storage Service (Amazon S3). You can now iterate over millions of objects such as logs, images, or. The new distributed map state allows you to write Step Functions to coordinate large-scale parallel workloads within your serverless applications. In order to achieve higher parallel processing prior to today, you had to implement complex workarounds to the existing map state component. This limit makes it challenging to scale data processing workloads to process thousands of items (or even more) in parallel. The existing map state is limited to 40 parallel iterations at a time. Step Function’s map state executes the same processing steps for multiple entries in a dataset. This flow extends support for orchestrating large-scale parallel workloads such as the on-demand processing of semi-structured data. I am excited to announce the availability of a distributed map for AWS Step Functions.
