Amazon Bedrock now supports batch inference
You can now use Amazon Bedrock to process prompts in batch to get responses for model evaluation, experimentation, and offline processing.
You can now use Amazon Bedrock to process prompts in batch to get responses for model evaluation, experimentation, and offline processing.
Using the batch API makes it more efficient to run inference with foundation models (FMs). It also allows you to aggregate responses and analyze them in batches.
Batch processing is available in preview in US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), and Europe (Frankfurt) AWS Regions.
To learn more about batch inference in Amazon Bedrock, see Amazon Bedrock API reference. Pricing for Batch mode is the same as pricing for On-Demand mode. For details, see the Amazon Bedrock pricing page.
Update: 2/27/2024 - The original post mistakenly listed the launch as generally available whereas it’s actually in preview, and has been updated accordingly.