SoftNAS Cloud NAS software paces itself when moving petabytes

SoftNAS engineers encountered a problem early this year with Amazon Web Services and Microsoft Azure clouds that caused a delay in the SoftNAS Cloud NAS 3.5 release.

The engineers hit a snag while doing quality assurance testing on pushing petabyte-scale migrations into the cloud using the SoftNAS primary storage filer. They needed to get the testing done in one month’s time, but the public cloud’s algorithms had other ideas.

“When you are moving massive amounts of data in the cloud, you have to have parallel I/O streams,” said Rick Braddy, CEO of SoftNAS. “When you do that, (the algorithms) start stiff-arming you and telling you to slow down. When you start sending data too fast, they start penalizing you. You get error messages and you have to do a retry.”

SoftNAS engineers hit a wall with the ingest rates. This problem forced them to innovate out of necessity because at the pace the cloud algorithms were dictating, the quality assurance testing would have taken six months to put a few petabytes into the cloud.

So the engineers developed patent-pending ObjFast technology, and integrated it into the company’s latest version of its SoftNAS Cloud NAS. The company claims the software-defined-storage product writes to the cloud more than twice as fast as previous SoftNAS Cloud NAS versions, effectively giving customers block-storage performance at the price of object storage.

“This basically delayed our whole product release by a quarter, so we ended up finishing the testing in the second quarter,” Braddy said.

Braddy said his team discovered in testing that each public cloud has its own pace for ingesting and the pacing varies depending on demand.

“It’s a shared service,” he said. “So the amount of throttling is based on everyone’s usage. What happens is if you don’t pace them correctly, they send you error messages and have to go into the retry loop.”

The new ObjFast technology uses massive parallel I/Os of object data streams coupled with algorithms to pace the data ingest rate for each I/O stream. That keeps the ingest rate from going over the clouds’ maximum allowed per stream data rate.

“We are not pacing their algorithm,” Braddy said. “We invented a proprietary pacing that adapts. Based on our testing, we found that Microsoft Azure’s overall ingest rate is faster than Amazon S3.”