AWS MAP for Storage Migration: Moving Petabytes to S3 and EFS

How Does AWS DataSync Handle Large-Scale Transfers?

AWS DataSync is a managed data transfer service that moves data between on-premises storage and AWS at speeds up to 10 Gbps. It supports NFS shares, SMB shares, HDFS clusters, self-managed object storage, and transfers between AWS storage services.

DataSync deploys an agent—either as a VMware ESXi, KVM, or Hyper-V virtual machine—in your data center. The agent connects to your source storage and transfers data to the destination AWS service. DataSync handles scheduling, bandwidth throttling, data integrity verification, and incremental transfers automatically.

Each DataSync task performs a comparison between source and destination. On the first run, it transfers all data. On subsequent runs, it identifies and transfers only changed or new files. This incremental approach means you can start the initial transfer weeks before cutover and run a final incremental sync that takes minutes instead of days.

DataSync charges per gigabyte transferred. For a petabyte migration, the DataSync cost is approximately $4,500 at standard pricing. MAP credits can offset this cost entirely. The service also compresses and encrypts data in transit, reducing actual bandwidth consumption by 30–50% for compressible file types.

For organizations also migrating databases alongside storage, coordinating with the database migration workstream ensures that application data and file data arrive on AWS in a consistent state.

When Should You Use the AWS Snow Family?

The Snow Family provides physical devices for offline data transfer. AWS ships a device to your data center, you load data onto it, and you ship it back. AWS imports the data into S3. This approach bypasses network bandwidth entirely.

AWS Snowball Edge Storage Optimized holds 80 TB of usable storage. For a 500 TB migration, you order seven devices, load them in parallel, and ship them back. AWS imports the data within days of receiving each device. Total elapsed time from ordering to data availability is typically 2–3 weeks—compared to months over the network.

AWS Snowcone holds 8 TB or 14 TB depending on the model. It weighs under 5 pounds and fits in a backpack. Snowcone suits remote office or edge location migrations where shipping a full-size Snowball Edge is impractical. It also works for locations with limited or no network connectivity.

AWS Snowmobile is a 45-foot shipping container that holds up to 100 PB. AWS drives it to your data center and connects it directly to your network. Snowmobile suits organizations migrating exabyte-scale data stores—typically media archives, scientific datasets, or financial record repositories. Few migrations need Snowmobile, but for those that do, no other option is feasible.

Snow devices charge a service fee per device per day plus data transfer costs on import to S3. MAP credits apply to these costs. The cost-per-TB is lower than DataSync for large volumes, but the lead time for ordering and shipping adds calendar days to the migration plan.

How Do You Choose Between S3, EFS, and FSx?

The target storage service determines performance characteristics, access patterns, and long-term costs. Selecting the right service during MAP planning prevents costly migrations between AWS services later.

Amazon S3 provides object storage with virtually unlimited capacity. It suits data lakes, backup archives, media repositories, and any workload that accesses data by key rather than file path. S3 offers storage classes from S3 Standard (frequent access) through S3 Glacier Deep Archive (long-term retention at $1 per TB per month). Lifecycle policies automate tiering between classes based on access patterns.

Amazon EFS provides managed NFS file storage. It suits Linux-based applications that need shared file system access. EFS scales automatically—you do not provision capacity. Multiple EC2 instances, Lambda functions, and ECS containers can mount the same EFS file system simultaneously. EFS costs more per GB than S3 but supports POSIX file system semantics that applications require.

Amazon FSx offers four managed file system options: FSx for Windows File Server (SMB), FSx for Lustre (high-performance computing), FSx for NetApp ONTAP (multi-protocol), and FSx for OpenZFS (Linux file serving). FSx for Windows File Server is the most common choice for Windows workload migrations. FSx for Lustre suits machine learning training and HPC workloads that need hundreds of thousands of IOPS.

Most organizations use a combination. Structured application data goes to EFS or FSx. Unstructured archives go to S3. Backups tier automatically through S3 storage classes. The MAP assessment phase should classify your storage estate by access pattern and map each category to the appropriate AWS service.

What Does a Petabyte-Scale Migration Plan Look Like?

Moving a petabyte or more requires combining multiple transfer methods. No single approach optimizes for cost, speed, and risk simultaneously. A proven pattern follows four stages.

Stage one is data classification and prioritization. Catalog all source storage: NAS arrays, SAN volumes, file servers, object stores, and tape archives. Classify by access frequency, business criticality, and target service. Identify data that can retire rather than migrate—most organizations find 20–40% of stored data is ROT (redundant, obsolete, trivial).

Stage two is the bulk initial transfer. For datasets over 100 TB, use Snow devices for the initial load. Order devices, load data, ship, and import. For datasets under 100 TB or when network bandwidth is sufficient, use DataSync for the initial transfer. Run both methods in parallel across different data sets to maximize throughput.

Stage three is incremental synchronization. Once the bulk load completes, configure DataSync tasks for ongoing synchronization. These incremental transfers capture changes made to source storage during the bulk load period. Run incremental syncs daily or hourly depending on change rates.

Stage four is cutover and validation. During the final cutover window, run a last incremental sync. Verify data integrity through checksum comparison—DataSync performs this automatically. Update application configurations to point to AWS storage endpoints. Decommission source storage after a retention period confirms the migration is complete.

How Does AWS Transfer Family Support Ongoing Operations?

AWS Transfer Family provides managed SFTP, FTPS, FTP, and AS2 endpoints that connect directly to S3 or EFS. It addresses the common post-migration requirement of accepting files from external partners who use legacy transfer protocols.

Many organizations receive data from suppliers, clients, or regulatory bodies via SFTP. On-premises, they run dedicated SFTP servers that store incoming files on NAS or SAN storage. After migrating storage to AWS, Transfer Family replaces those SFTP servers. Partners continue uploading to the same DNS endpoint. Files land directly in S3 buckets or EFS file systems.

Transfer Family supports identity federation through Active Directory, LDAP, or custom Lambda-based authentication. Existing SFTP user credentials work without changes. This continuity is essential—you cannot ask hundreds of external partners to update their SFTP configurations simultaneously.

For MAP funding purposes, Transfer Family consumption during the migration period counts as eligible AWS spending that MAP credits can offset.

What Are Common Pitfalls in Storage Migration?

Permission mapping failures cause the most post-migration issues. On-premises NFS exports with UID/GID-based permissions need equivalent mappings on EFS. Windows NTFS ACLs require Active Directory integration on FSx. Plan permission mapping during the mobilization phase and validate with test transfers before production cutover.

Underestimating network bandwidth requirements stalls timelines. If your DataSync transfers compete with production traffic on the same WAN link, both suffer. Dedicate bandwidth for migration traffic or use AWS Direct Connect for a dedicated path. A 10 Gbps Direct Connect circuit costs approximately $2,250 per month and transfers roughly 3 PB per month at full utilization.

Ignoring application dependencies creates cutover failures. Applications that reference UNC paths (\\server\share) or NFS mount points need configuration updates. Identify every application that references migrating storage and plan configuration changes for the cutover window. DNS aliasing can reduce the number of application changes required—point the old server name to the new FSx or EFS endpoint.

Forgetting about tape archives leaves data behind. Many organizations have years of backup data on tape that needs to migrate to S3 Glacier. AWS Tape Gateway can present S3 Glacier as a virtual tape library, but tape ingest throughput is limited by drive speed. Factor tape migration time into your overall timeline.

Conclusion

Storage migration under AWS MAP is a high-volume, high-value workstream that requires methodical planning and the right combination of transfer tools. DataSync handles online transfers efficiently. The Snow Family bypasses bandwidth constraints for large datasets. S3, EFS, and FSx each serve different access patterns, and selecting the right target prevents costly re-migration later. MAP credits offset the transfer costs, device fees, and AWS consumption that make large-scale storage moves expensive. Start with data classification, eliminate ROT data, and execute transfers in parallel to minimize the overall migration window.