New enhancements for moving data between Amazon FSx for Lustre and Amazon S3
Amazon FSx for Lustre, a high performance file system optimized for workloads such as machine learning, high performance computing, video processing, financial modeling, electronic design automation, and analytics, has added functionality that makes it easier to synchronize file data and file permissions between Amazon FSx and Amazon S3. Additionally, Amazon FSx has quadrupled the speed of launching FSx file systems that are linked to S3 buckets.
Amazon FSx for Lustre, a high performance file system optimized for workloads such as machine learning, high performance computing, video processing, financial modeling, electronic design automation, and analytics, has added functionality that makes it easier to synchronize file data and file permissions between Amazon FSx and Amazon S3. Additionally, Amazon FSx has quadrupled the speed of launching FSx file systems that are linked to S3 buckets.
Amazon FSx for Lustre works natively with Amazon S3, making it easy for you to process cloud data sets with high performance file systems. When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write results back to S3. Until today, Amazon FSx provided file system commands to export new or changed files from FSx to S3, but these commands did not copy file permissions or provide the ability to monitor or cancel the transfer.
Today, Amazon FSx for Lustre is launching an AWS application programming interface (API) that allows you to easily export files from FSx to S3. With the new API, you can easily initiate, monitor, and cancel writing new or changed files to S3. Since it’s an AWS-native API, you can use it to easily orchestrate data export tasks from cloud native workflows such as Lambda-based serverless applications. In addition to transferring file data and file permissions, this API also allows you to transfer symbolic links, file ownership metadata, and file time stamps to S3. File permissions and other file metadata are stored in S3 in the same format used by AWS DataSync and AWS Storage Gateway.
Additionally, FSx has quadrupled the speed at which file metadata is imported from S3, allowing you to launch S3-backed FSx file systems up to four times faster.
The new functionality, API, and performance improvements are available on all new file systems in all regions where FSx for Lustre is available. For more information, please visit the Amazon FSx for Lustre product page, and check out our blog post here.