S3 features unveiled as Amazon reflects on object storage past
Amazon introduced new features to its storage platform that increase speed and container options during a webinar today. It also looked back at S3's launch 18 years ago.
Eighteen years after the launch of Amazon's Simple Storage Service, there's still plenty to do to keep the enterprise platform current, an AWS storage executive said.
Three new S3 object storage services and a speed bump to the Amazon Elastic File Service (EFS) enable developers to work with AI and ML workloads, said Andy Warfield, vice president and distinguished engineer at AWS Storage.
If there's any time a developer has to think about storage or about how they're going to make an application talk about their data, that's a miss on our part.
Andy WarfieldVice president and distinguished engineer, AWS
"If there's any time a developer has to think about storage or about how they're going to make an application talk about their data, that's a miss on our part," Warfield said.
The new S3 capabilities include save checkpoints for PyTorch Lightning directly to S3, authentication caching for Amazon S3 on Outposts to reduce network traffic and an S3 Container Storage Interface (CSI) driver for the Bottlerocket Linux container OS on AWS.
Throughput for Amazon EFS has doubled, now offering reads of 20 gibibytes per second and 5 GiBps writes, according to the vendor.
These additions to S3 are aimed at minimizing the time it takes developers to consider storage implementation, Warfield said.
Object storage on the S3 open source standard, widely adopted in both enterprise clouds and data centers, will only gain additional complexity and features to meet AI demands, according to Simon Robinson, an analyst at TechTarget's Enterprise Strategy Group.
"Object is evolving from cheap and deep to more for the AI age," he said.
Simple Storage Service(s)
The Amazon S3 on Outposts authentication caching eliminates the need for data to authenticate with a parent AWS Region for every request. This service enables the hardware to cache authentication and authorizations locally for a set time. Outposts is an on-premises IT-as-a-service platform that provides cloud services through local hardware.
The Amazon S3 Connector for PyTorch, released last November, connects S3 directly to jobs using the PyTorch open source machine learning framework. The new capability, available today, adds support for saving model checkpoints without pausing ongoing jobs from PyTorch Lightning, a variant of PyTorch that targets large language models and generative AI workloads.
The Mountpoint for Amazon S3 CSI driver is now supported in Bottlerocket, making S3 objects accessible for containers in Amazon Elastic Kubernetes Service or self-managed Kubernetes clusters with a file system interface.
The bump in speed to Amazon EFS is a sign that the vendor is continuing to get serious about enterprise file storage systems, Robinson said. But it still has a way to go to catch up to its partner NetApp, he added. NetApp and Amazon's combined offering, Amazon FSx for NetApp OnTap, improved throughput last month, increasing speeds from 36 GBps to 72 GBps.
NetApp has been a dominant player in enterprise storage for decades, Robinson said, so any born-in-the-cloud storage product will likely lag behind established on-premises players.
"To do enterprise file at scale is extraordinarily difficult," he said. "The reason NetApp is so dominant is because they've been doing it for 30 years. AWS is playing catch-up here."
NetApp isn't alone as a data center storage vendor eyeing cloud dominance, according to Ray Lucchesi, president and founder of Silverton Consulting.
"Dell Technologies, Hitachi Vantara and IBM are all looking at the cloud as the next great revenue stream -- and they're not standing still," he said.
Baking in more
The new releases, which coincide with AWS Pi Day 2024, also focus on how the hyperscaler is becoming more malleable for enterprise generative AI strategies, Warfield said. The vendor's push to do so should eventually bleed into enterprise services related to storage.
"Customers want as much flexibility as they can have [with storage]," Warfield said. "We're trying to give them the APIs they need so they don't have to do extra work to adapt to a specific type of storage."
The embrace of S3 and its associated APIs over time enabled adoption of object storage in applications and enterprise uses, Robinson said, paving the way for AWS to grow as customers sought to take data to the cloud.
Other vendors and storage companies are pitching alternatives to object storage for certain workloads, such as Hammerspace's new NFS offering, but object will likely remain the storage infrastructure of the cloud, he said.
"I don't think it's overstating it to say S3 changed the storage landscape completely," Robinson said. "Not just the storage architecture, but the application landscape as well."
S3 made object storage, a previously niche data center storage format, into a more easily accessible and cheap form of storage for cloud, Lucchesi said. The language to interact with S3, with get and put requests, mirrored how to interact with the internet at the time.
"The question was, what are you going to use [object storage] for?" he said. "S3 has become ubiquitous now. Object storage for a long time was looking for a killer app -- and S3 became that."
Tim McCarthy is a news writer for TechTarget Editorial covering cloud and data storage.
