AWS S3 Migration with DataSync: Fast & Easy Cross-Account S3 Transfers

AWS DataSync: Effortless S3 Migration Across AWS Accounts

Imagine you have a huge amount of data in your S3 bucket — maybe gigabytes (GB) or even petabytes (PB) — with multiple folders and objects. You need to move it all to another S3 bucket in a different AWS account. Sounds like a big task, right? 😓

Manually downloading, organizing, and re-uploading everything would take a very long time! ⏳ But don’t worry — AWS has a solution! 🚀 AWS DataSync makes this process super simple and fast, transferring all your data in just a few seconds. No more manual work — just quick, easy, and secure migration! ✅

What is AWS DataSync?

AWS DataSync is a cloud service that makes data migration easy. It helps you quickly, securely, and efficiently transfer files or object data to, from, and between AWS storage services without the hassle of manual work.

In this blog, we’ll explore how you can move all data from one S3 bucket to another S3 bucket in a different AWS account using AWS DataSync.

Prerequisites

Before we begin, ensure we have the following:

1. Source S3 Bucket — The bucket containing the data you want to migrate.
2. Destination S3 Bucket — The bucket where the data will be transferred.
3. AWS CLI Configured — The AWS CLI should be set up for the destination AWS account with the access key and secret key.

Follow these steps to migrate all S3 data from one AWS account to another using AWS DataSync:

Step-1: Create DataSync IAM Role in destination S3 AWS account.

• Go to IAM → Roles in the AWS Management Console.
• Click on Create Role.
• Trusted entity type → Select AWS Service.
• Use case → Choose DataSync.

DataSync IAM Role

• Click Next, then Next again, and create the role with the name “DatasyncRole”.
• Open the newly created DatasyncRole and add the following inline policy with the name “DatasyncRole-migration-policy”.

Note: Make sure to replace it <source_bucket> with the name of your actual source S3 bucket.

Step 2: Add a Bucket Policy in the Source AWS Account

In the source AWS account, update the S3 bucket policy to allow AWS DataSync to access and migrate data.

Note: Replace the placeholders in the bucket policy with your actual values:

• <DatasyncRole-ARN> → The ARN of the IAM role created for DataSync in the destination AWS account.
• <USER_NAME> → Your IAM username in the destination AWS account.
• DIST_ACCOUNT_ID → The AWS account ID of the destination account.
• <SOURCE_BUCKET_ARN> → The ARN of your source S3 bucket.

Make sure you use <USER_NAME> that access key and secret key is used in aws configure for destination account locally as mentioned in preresuiesties.

You can get DIST_ACCOUNT_IDand <USER_NAME>using the following query command:

aws sts get-caller-identity --query "Account" --output text
aws iam get-user --query "User.UserName" --output text

Step 3: Add a Bucket Policy in the Destination AWS Account

In the destination AWS account, update the S3 bucket policy to allow AWS DataSync to write data to this bucket.

Note: Replace the placeholders in the bucket policy with your actual values:

• <DatasyncRole-ARN> → The ARN of the IAM role created for DataSync in the destination AWS account.
• <DestinationBucket-ARN>→ The ARN of your destination S3 bucket.
• DIST_ACCOUNT_ID → The AWS account ID of the destination account.
• <USER_NAME> → Your IAM username in the destination AWS account.

Here <USER_NAME> and DIST_ACCOUNT_ID same as in step 2.

Step 4: Disable ACLs on the Destination S3 Bucket

1. Open the S3 console and select the destination bucket where you’re transferring data.
2. Go to the Permissions tab.
3. Scroll to Object Ownership and click Edit.
4. Select ACLs disabled (recommended) if it’s not already selected.
5. Click Save Changes.

Step 5: Configure Location Point for DataSync on the Destination AWS Account using AWS CLI

In AWS DataSync, we have two location points:

1. Source Location — The S3 bucket from which data is migrated.
2. Destination Location — The S3 bucket where data is transferred.

• In our case, we are creating the Source location using the AWS CLI.

To create a source location in the destination AWS account, run the following command:

aws datasync create-location-s3 --s3-bucket-arn <Source_Bucket_ARN> \
  --s3-storage-class STANDARD \
  --s3-config BucketAccessRoleArn="<DatasyncRole-ARN>" \
  --region <REGION>

Note: Replace the placeholders in the bucket policy with your actual values:

• <Source_Bucket_ARN> → ARN of the source S3 bucket.
• <DatasyncRole-ARN> → ARN of the IAM role created for DataSync that is created in the destination AWS account.
• <REGION> → AWS region where the destination bucket is located.

Once you create the source location, you will receive an output similar to this:

DataSync destination location

In the Destination AWS account, navigate to AWS DataSync → Locations, and you will see a newly created location with the source S3 bucket as the host or source location.

DataSync destination location

For the destination location, we don’t need to create it separately. During the DataSync setup, we can manually select the source S3 bucket from which we are migrating the data.

Step 6: Create a Task on DataSync for S3 Migration

1. Go to destination AWS DataSync service → Tasks
2. Click on “Create Task”
3. In Source Location:

• Choose “Use an existing location.”
• Select the source location (your source S3 bucket).

Source Location Configuration in AWS DataSync

• Tap next.

4. Configure Destination Location in DataSync

• Select “Create a new location.”
• Use the below Configuration:
• Location Type: Amazon S3
• Select Region (same as the destination S3 bucket)
• Select S3 Bucket (this is your destination S3 bucket)
• S3 Storage Class: Standard
• IAM Role: Select the role created in Step 1 (DatasyncRole)

Destination Location Configuration in AWS DataSync

5. Final Step: Configure Task Settings

• In the last stage of task creation, configure the following:
• Task Mode: Basic
• Task Name: Enter a name for your task.
• Logging:
• Log Level: Select “Log basic information such as transfer errors.”
• CloudWatch Log Group: Choose “Autogenerate.”
• Review and Create the Task.

Final DataSync Task Configuration

Now, our DataSync task is ready to migrate S3 data from the Source AWS account to the Destination AWS account.

Step 7: Start S3 Migration with DataSync

• Go to AWS DataSync → Your Created Task
• Click Start
• Select “Start with Default” settings
• Confirm & Begin Migration 🚀

AWS DataSync S3 Data Migrtion

Step 8: AWS DataSync — Final Migration Result

Once the task is completed, you can verify that all data has been migrated from the Source AWS account S3 to the Destination AWS account S3. 🎉

✅ Check your destination S3 bucket to confirm the data transfer! 🚀

AWS DataSync Task Complted

Destination AWS S3 Bucket

Connect With Me

Catch me on LinkedIn for more insights and discussions! Together, let’s navigate the intricate world of AWS, cloud strategies, Kubernetes, and beyond. Connect with me to exchange ideas, seek advice, or simply to say hello.

Happy Migrating! 🚀

Happy Learning! 📚