AWS Glue Data Catalog now supports PartitionIndex, improving query performance on highly partitioned tables
AWS Glue Data Catalog now supports PartitionIndex on tables. As you continually add partitions to tables, the number of partitions can grow significantly over time causing query times to increase. With PartitionIndexes, you can reduce the overall data transfers and processing, and reduce query processing time.
AWS Glue Data Catalog now supports PartitionIndex on tables. As you continually add partitions to tables, the number of partitions can grow significantly over time causing query times to increase. With PartitionIndexes, you can reduce the overall data transfers and processing, and reduce query processing time.
PartitionIndex can be created at the time of table creation by specifying an ordered list of partition keys that already exist on a given table. The AWS Glue Data Catalog will create a fast, searchable index based on these keys.
PartitionIndex is available in all regions where AWS Glue is available.
To learn more, visit the AWS Glue Developer Guide.