Amazon S3 server access logging now supports automatic date-based partitioning
Amazon S3 server access logging now supports automatic date-based partitioning for log delivery. Amazon S3 server access logging provides detailed records for requests made to your S3 buckets including object size, total time, turn-around time, HTTP referer, and more. Now, with date-based partitioning, Amazon S3 automatically generates either event time or delivery time prefixes when delivering access logs to your destination bucket, which allows services like Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum to improve performance and reduce cost when querying logs.
Amazon S3 server access logging now supports automatic date-based partitioning for log delivery. Amazon S3 server access logging provides detailed records for requests made to your S3 buckets including object size, total time, turn-around time, HTTP referer, and more. Now, with date-based partitioning, Amazon S3 automatically generates either event time or delivery time prefixes when delivering access logs to your destination bucket, which allows services like Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum to improve performance and reduce cost when querying logs.
Date-based partitioning improves performance and cost-efficiency of downstream log processing systems by limiting the scanning of logs to only the desired time range. For example, with Amazon Athena’s ability to read date-based partitioned logs directly from S3, it’s more performant and cost-efficient to query for all operations that were performed on an object within a specific time period, or to identify all requests that required an ACL for authorization within a specific time period.
Automatic date-based partitioning for Amazon S3 server access logging is available at no additional charge in all AWS Regions, including the AWS GovCloud (US). To get started, enable date-based partitioning on your server access logging configuration using the Amazon S3 console, AWS Command Line Interface (CLI), Amazon S3 REST API, or AWS Software Development Kits (SDKs). To learn more, refer to the S3 server access logging documentation.