aws-s3-sync

Sync files and directories from local to a bucket

$ aws s3 sync [path/to/file_or_directory] s3://[bucket_target_name]/[path/to/remote_location]

$ aws s3 sync s3://[bucket_source_name]/[path/to/remote_location] [path/to/file_or_directory]

$ aws s3 sync s3://[bucket_source_name]/[path/to/remote_location] s3://[bucket_target_name]/[path/to/remote_location]

$ aws s3 sync [path/to/file_or_directory] s3://[bucket_target_name]/[path/to/remote_location] --exclude [path/to/file] --exclude [path/to/directory]/*

$ aws s3 sync s3://[bucket_source_name]/[path/to/remote_location] s3://[bucket_target_name]/[path/to/remote_location] --delete

$ aws s3 sync [path/to/local_directory] s3://[bucket_name]/[path/to/remote_location] --acl [private|public-read] --storage-class [STANDARD_IA|GLACIER]

$ aws s3 sync [path/to/file_or_directory] s3://[bucket_name]/[path/to/remote_location] --size-only

$ aws s3 sync help

aws s3 sync source_path destination_path [options]

source_path
    The path to the source directory or S3 prefix (e.g., `s3://bucket/prefix`). This can be a local directory path (e.g., `/path/to/local/directory`).

destination_path
    The path to the destination directory or S3 prefix. This can be a local directory path or an S3 path (e.g., `s3://bucket/prefix`).

--delete
    Deletes files in the destination that do not exist in the source. Use with caution, as this can result in data loss.

--exclude pattern
    Excludes files or directories matching the specified glob-style pattern from the synchronization. Can be specified multiple times.

--include pattern
    Includes files or directories matching the specified glob-style pattern in the synchronization. Overrides excludes. Can be specified multiple times.

--acl permission
    Sets the Access Control List (ACL) permissions for uploaded objects (e.g., `public-read`, `private`).

--dryrun
    Performs a dry run, showing the actions that *would* be taken without actually executing them.

--quiet
    Suppresses output. Only errors are displayed.

--storage-class storage_class
    Sets the storage class for uploaded objects (e.g., `STANDARD`, `REDUCED_REDUNDANCY`, `GLACIER`).

--sse encryption_type
    Enables server-side encryption (SSE) using the specified encryption type (e.g., `AES256`, `KMS`).

--sse-kms-key-id key_id
    Specifies the KMS key ID to use for SSE-KMS encryption.

--follow-symlinks
    Follows symbolic links when synchronizing files. By default, symlinks are not followed.

--only-show-errors
    Shows only errors

--page-size int
    The number of results to return in each page of the listing.

--no-progress
    Do not display the progress of an operation.

The `aws s3 sync` command is a powerful tool within the AWS Command Line Interface (CLI) for synchronizing directories between a local file system and an Amazon S3 bucket. It efficiently copies files, ensuring that the target directory (either local or S3) mirrors the source. This includes transferring new files, updating modified files, and optionally deleting files in the target that no longer exist in the source. It's particularly useful for tasks like backing up local data to S3, deploying website content from a local directory to an S3 bucket configured for static website hosting, or maintaining consistent data between a development environment and a cloud storage location. Key benefit: Only transfers changes, minimizing bandwidth and time.

The command employs checksums to determine which files need to be transferred, optimizing performance. Options exist to control recursion (walking subdirectories), exclude/include specific files or patterns, set access control lists (ACLs), and configure other aspects of the synchronization process. Effectively you have a powerful replication command for synchronising the file system with an AWS S3 storage.

Incorrectly configured IAM permissions or S3 bucket policies can prevent the command from working. `--delete` option should be used with extreme caution to prevent accidental data loss. Large synchronizations can take a significant amount of time and bandwidth.

• Sync a local directory to an S3 bucket:
  aws s3 sync /path/to/local s3://my-bucket/my-prefix
• Sync an S3 bucket to a local directory:
  aws s3 sync s3://my-bucket/my-prefix /path/to/local
• Sync with deletion:
  aws s3 sync /path/to/local s3://my-bucket/my-prefix --delete
• Exclude specific files:
  aws s3 sync /path/to/local s3://my-bucket/my-prefix --exclude "*.tmp" --exclude "temp/*"

The command returns a non-zero exit code if any errors occur during the synchronization process. Inspect the output for error messages. Check AWS credentials and bucket permissions if you encounter authorization errors.

The `aws s3 sync` command was introduced as part of the AWS CLI to simplify and automate the process of synchronizing data between local file systems and Amazon S3. It has become a widely used tool for managing data in S3, especially in deployment pipelines and backup scenarios. Over time, its features have been expanded to include support for encryption, ACLs, and other S3 features.

aws s3 cp(1), aws s3 mv(1), aws s3 rm(1), rsync(1)