Amazon Kinesis Data Streams announces two new API features to simplify consuming data from Kinesis streams
Amazon Kinesis Data Streams announces two new API features that simplify applications that consume data from Kinesis Data streams. The first feature is an optional filtering field in the ListShards API that enables you to set filtering criteria for the shards that you want to list from a data stream. For example, you can now use a timestamp filter to list shards with data at a certain time point. The second feature is a new field called ChildShards in the GetRecords and the SubscribeToShards APIs that enables you to discover child shards of a closed shard (without having to use the ListShards API to list all closed and open shards). Applications consuming data from Kinesis Data Streams can use this new field to efficiently transition reading from a closed shard to reading from its child shard. This is a common use case encountered by consuming applications when a shard is closed due to stream scaling operations.
Amazon Kinesis Data Streams announces two new API features that simplify applications that consume data from Kinesis Data streams. The first feature is an optional filtering field in the ListShards API that enables you to set filtering criteria for the shards that you want to list from a data stream. For example, you can now use a timestamp filter to list shards with data at a certain time point. The second feature is a new field called ChildShards in the GetRecords and the SubscribeToShards APIs that enables you to discover child shards of a closed shard (without having to use the ListShards API to list all closed and open shards). Applications consuming data from Kinesis Data Streams can use this new field to efficiently transition reading from a closed shard to reading from its child shard. This is a common use case encountered by consuming applications when a shard is closed due to stream scaling operations.
Kinesis Client Library (KCL) helps you quickly build custom consumer applications by handling complex issues such as adapting to changes in stream volume, load-balancing streaming data, coordinating distributed workers, and processing data with fault-tolerance. KCL enables you to focus on business logic while building consumer applications. Customers using the latest KCL versions, KCL 1.14 for standard consumers and KCL 2.3 for EFO consumers, will automatically benefit from these two new features.
Amazon Kinesis Data Streams is a massively scalable and durable real-time data streaming service. It can continuously capture gigabytes of data per second from hundreds of thousands of sources like website clickstreams, IoT data, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. The data collected is available in milliseconds for real-time analytics use cases like dashboards, anomaly detection, dynamic pricing, and more.
For more information, refer to Reading data from kinesis data streams in the Amazon Kinesis Data Streams Developer Guide and API Reference for ListShards, GetRecords, and SubscribeToShard. These API enhancements and latest KCL releases are available in all AWS regions where Amazon Kinesis Data Streams is available.