Amazon Redshift announces public preview of Streaming Ingestion for Kinesis Data Streams
Amazon Redshift launches Streaming Ingestion support for Kinesis Data Streams (KDS). Amazon Redshift’s Streaming Ingestion eliminates the need to stage data in Amazon S3 before ingesting it into Amazon Redshift, enabling customers to achieve low latency in the seconds while ingesting hundreds of megabytes of streaming data per second into their data warehouse.
Amazon Redshift launches Streaming Ingestion support for Kinesis Data Streams (KDS). Amazon Redshift’s Streaming Ingestion eliminates the need to stage data in Amazon S3 before ingesting it into Amazon Redshift, enabling customers to achieve low latency in the seconds while ingesting hundreds of megabytes of streaming data per second into their data warehouse.
Data engineers, data analysts, and big data developers are evolving their analytics from batch to real-time, adopting streaming engines like Amazon Kinesis, to implement near real-time responsive logic and analytics on streaming application data. Currently, customers who want to ingest real-time data, from services like Amazon Kinesis, into Amazon Redshift must first stage the data in Amazon S3 and use the COPY command, which achieves latency in the minutes. With the new streaming ingestion capability in Amazon Redshift, you can use SQL (Structured Query Language) within Redshift to provide the ability to connect to and directly ingest data from multiple Kinesis data streams simultaneously. Amazon Redshift streaming ingestion simplifies data pipelines by letting you create materialized views on top of streams directly. The materialized views can also include SQL transforms as part of your ELT (Extract Load Transform) pipeline. Once the materialized views are defined, you can manually refresh them to allow you to query the most recent stream data. This approach allows you to perform downstream processing and transformations of streaming data using existing tools and SQL that you are familiar with, at no additional cost.
Amazon Redshift Streaming Ingestion preview is now available in all Amazon Web Services (AWS) regions where Amazon Redshift is available. To get started with Amazon Redshift Streaming Ingestion, provision an Amazon Redshift cluster on the CURRENT Track and verify your cluster is running version 1.0.35480 or newer. Please note, preview features are provided primarily for evaluation and testing purposes. They should not be used in production systems or with production data. For preview terms and conditions, see Beta Service Participation in AWS Service Terms. For more information on how to get started with Amazon Redshift Streaming Ingestion, see the Amazon Redshift documentation page. Please do not hesitate to contact us with questions or feedback at: redshiftstreamingingestion@amazon.com.