Data migration specialist Datadobi adds S3-to-S3 support

Data migration from AWS

But Michael Jack, Datadobi's co-founder and VP of global sales, expects the most popular use case could be AWS to on-premises object storage to help customers avoid the egress fees that Amazon and other cloud providers charge to access data. A secondary source of demand could be customers refreshing their object storage for better functionality or higher performance.

"There was a big flurry of media a year and a half ago around repatriation -- that repatriation is going to be huge because Amazon's too expensive," Jack said. But, rather than a "massive tsunami of opportunities," Jack expects to see "incremental business" from customers that don't want to be locked in to a particular storage vendor or storage.

A lot of people get upset about the egress fees that they didn't really understand with public cloud storage.

"A lot of people get upset about the egress fees that they didn't really understand with public cloud storage," said Marc Staimer, president of Dragon Slayer Consulting. "They want to move, but there's no easy way to migrate from S3 to S3 natively, so they need a third party."

Datadobi specializes in migrating large quantities of data. The company initially focused on moving data out of EMC's Centera content-addressable storage and then built DobiMigrate to tackle NAS migration when EMC's Isilon team reached out. The DobiMigrate 5.9 update will add connectors to S3 on top of the NAS migration framework, incorporating important lessons that Datadobi's engineering team learned through its NAS work about the incompatibilities and vagaries in different vendors' systems.

S3 data migration is not trivial

"We thought S3 would be simple, because it's a simple protocol, but the same is true here," D'Halluin said. "Copying over S3 data is not trivial, and enterprises want some guarantees that the data is copied over properly."

D'Halluin said Datadobi engineers had to "invent new ways" to scan the S3 content and enable fast copying and parallel scanning. DobiMigrate runs on premises behind the data center's firewall, under the control of the data center administrator. The software can synchronize the objects and metadata from the source buckets to the target buckets.

"One of the product's values to the enterprise is the scale at which it can operate," said Krista Macomber, a senior analyst at Evaluator Group. "This is important for enterprises looking to migrate petabytes of data. The scanning process is multithreaded, and initial full, as well as incremental, scans occur outside of the data path. This helps to accelerate the scanning and copying process and cut down on the maintenance window."

Macomber thinks more Datadobi customers could move S3 data to the cloud, for disaster recovery or archival purposes or as part of a legacy application migration. But she said pulling data back on premises could also be important as business analytics becomes more popular.

DobiMigrate supports only NAS-to-NAS or S3-to-S3 migrations. The software does not offer gateway services to migrate file-based NAS data to S3-based object stores, or vice versa. Datadobi supports the migration of data from NAS or Centera systems to S3 object stores only on an individual project basis.

Datadobi roadmap

Datadobi roadmap items include optimizing migrations from Amazon's Glacier, cold storage as well as support for advanced object features such as locking and tagging. D'Halluin said there is no incentive yet to support migrations between S3-based object stores and Microsoft's Azure because product support for Azure's Blob object storage API is minimal.

D'Halluin said Datadobi plans to phase in S3-to-S3 migration capabilities and help customers analyze which data to bring back from AWS, keep in the cloud or delete altogether. He expects customers initially will need help, so Datadobi will offer the new S3-to-S3 functionality as a service before making it available through the do-it-yourself software.

DobiMigrate has a 50 TB minimum for S3-to-S3 migrations. Jack estimated the service list price would start at about $45,000 for a 50 TB migration, depending on the customer's choice of storage vendor, systems integrator or a value-added reseller to provide it. Jack said the do-it-yourself software would list at about $30,000 for an S3-to-S3 migration of 50 TB of data.

Jack estimated that 25% of Datadobi's customers use the software on their own, and 75% go through the company's more than 100 partners. Datadobi sells only through the channel, but Jack said the company's sales organization communicates directly with all customers. In addition to DobiMigrate, Datadobi sells DobiReplicate for NAS-to-NAS file-based replication and DobiSync for file-to-object synchronization.

Datadobi claims to have more than 800 customers, including service providers and enterprises in industries such as financial services, healthcare, oil and gas, and media and entertainment. They tend to have large amounts of data that can scale past 1 PB in complex multivendor environments spanning on-premises and cloud sites, according to Jack.

Datadobi's competition includes free tools such as robocopy and rsync, as well as Data Dynamics and Komprise. Jack said Datadobi focuses on large one-time, "fire-hose" types of migrations in comparison to the "trickle" approach that Data Dynamics and Komprise take with their data management/migration products.

Staimer said StrongBox Data Solutions offers S3-to-S3 migration as a feature but finds that the quantities of moved object-based data tend to be small in comparison to the multi-petabyte projects the vendor does with file-based data.