AWS DataSync adds filtering for data transfers
When using AWS DataSync, you now have the option of configuring file path and object key filters to control the data copied between your NFS servers, Amazon S3 buckets, and Amazon EFS file systems. Filters enable you to specify the files, folders, or objects in the source location that you want your DataSync task to copy. With filters you can easily split a single file system between multiple storage destinations, exclude copying temporary files and folders, or copy only a subset of files from your source location.
When using AWS DataSync, you now have the option of configuring file path and object key filters to control the data copied between your NFS servers, Amazon S3 buckets, and Amazon EFS file systems. Filters enable you to specify the files, folders, or objects in the source location that you want your DataSync task to copy. With filters you can easily split a single file system between multiple storage destinations, exclude copying temporary files and folders, or copy only a subset of files from your source location.
You can specify an exclude filter, an include filter, or both, to limit which files, folders or objects gets transferred each time a task runs. When creating a task, you configure the file paths or object keys that should always be excluded from being copied. Then, when you start a task, you configure the file paths or object keys that should be included for that execution of the task. If no filters are configured, each time a task runs it will transfer all changes from the source to the destination.
Filtering is available in all regions where AWS DataSync is available. You can learn more by reading our blog on using common filters with DataSync, the DataSync documentation, or log in to the AWS DataSync console to get started.