How to Copy S3 Bucket Objects Across AWS Accounts

In this blog, we’ll explore the process of copying S3 bucket objects across different AWS accounts. This task is crucial for scenarios such as data migration, backups, or simply sharing resources between departments or projects within your organization. As cloud solutions evolve, it’s increasingly common to manage multiple AWS accounts. Understanding how to transfer S3 objects securely and efficiently between these accounts can be a valuable skill.

Prerequisites

Before you start, make sure you have:

Access to two AWS accounts – the source and the destination.
Permissions to read objects from the source S3 bucket.
Permissions to write objects into the destination S3 bucket.
AWS CLI installed and configured on your machine. Alternatively, you can use the AWS Management Console.
The names of both the source and destination buckets.

Step 1: Prepare the Source Account

Set up the correct permissions in the source account so that the destination account can access the S3 objects.

Update the Bucket Policy

Firstly, you need to configure the necessary permissions. On the source account:

Navigate to the Amazon S3 console.
Select the bucket containing the objects to copy.
Go to the Permissions tab and edit the bucket policy to grant read access to the destination account.

Example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::<destination-account-id>:root"
          },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<source-bucket-name>/*",
                "arn:aws:s3:::<source-bucket-name>"
            ]
        }
    ]
}

Replace destination-account-id with the actual destination AWS account ID and source-bucket-name with the name of the source S3 bucket.

Step 2: Create an IAM Role in the Destination Account

With permissions set, now switch to the destination account and use the AWS CLI to copy the objects:

Set IAM Role & Trust Policy

Create an IAM role in the destination account that trusts the source account and has the permissions to copy objects from the source bucket.

Add the trust policy below that allows the role to be assumed by entities in the source account:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::<source-account-id>:root"
    },
    "Action": "sts:AssumeRole"
  }]
}

Replace source-account-id with the source AWS account ID.

Permissions Policy Attached to the Role

Attach a permissions policy that enables copying objects to the destination bucket.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:ListBucket"
        ],
        "Resource": "*"
    }]
}

Step 3: Assume the IAM Role from the Destination Account

Use the credentials of the destination account to assume the IAM role created earlier. This provides the security context to access resources from the source account.

Command to Assume Role

aws sts assume-role --role-arn arn:aws:iam::destination-account-id:role/role-name --role-session-name session-name

Remember to replace role-arn with the ARN of the IAM role created in the destination account.

Please note that you can run this command on your local machine, whether it’s Ubuntu or Linux WSL machine, as long as your AWS credentials are configured for your destination account. Here’s a guide on how to set up your AWS credentials on your local machine.

Step 4: Copy S3 Objects Using AWS CLI

With the permissions set up, use the AWS CLI to perform the copy operation from the source to the destination bucket.

AWS CLI Copy Command

aws s3 cp s3://source-bucket-name/object-key s3://destination-bucket-name/object-key --source-region source-region --region destination-region --profile assumed-role-profile

You can also use the --recursive option to copy all objects or specify multiple keys for individual objects.

Step 5: Verify the Transfer

After copying, it’s good practice to verify that the objects have been transferred correctly.

List Objects in the Destination Bucket

aws s3 ls s3://destination-bucket-name --profile destination-profile

Ensure that the list matches the expected objects in the destination bucket.

Conclusion

Copying S3 bucket objects across AWS accounts might seem challenging at first, but by following the steps outlined above, you will find that the process is quite straightforward. We discussed setting up the necessary permissions, using the AWS CLI to perform the copy operation, and validating the success of your actions. By mastering this technique, you can easily manage data across multiple AWS environments.

Whether you’re doing migrations, creating backups, or sharing data between teams, knowing how to move S3 objects between accounts is a critical tool in your AWS toolkit. For additional details and more complex scenarios, reference the official AWS guide here.