Amazon tech VP lays out ambitious AWS storage vision

LAS VEGAS --There appears to be no end in sight to the ambitious vision of AWS storage, especially when it comes to file systems.

During an interview with TechTarget, Amazon VP of technology Bill Vass said AWS aims to "enable every customer to be able to move to the cloud." For example, Amazon could offer any of the approximately 35 file systems that its enterprise customers use, under the FSx product name, based on customer demand, Vass said. FSx stands for File System x, where the "x" can be any file system. AWS launched the first two FSx options, for Lustre and Windows file systems, at its November 2018 Re:Invent conference.

(Editor’s note: Vass said during the original interview that AWS will offer all 35 file systems over time. After the article published, Vass contacted us via email to clarify his statement. He wrote: “FSx is built to offer any type of file system from any vendor. I don’t want it to seem that we have committed to all 35, just that we can if customers want it.”)

AWS cannot support nearly three dozen file systems overnight, but Vass highlighted a new storage feature coming in 2020: a central storage management console similar to the AWS Backup option that unifies backups.

Vass has decision-making oversight over all AWS storage products (except Elastic Block Storage), as well as quantum computing, IoT, robotics, CloudFormation, CloudWatch monitoring, system management, and software-defined infrastructure. Vass has held CEO, COO, CIO, CISO and CTO positions for startups and Fortune 100 companies, as well as the federal government. Before joining Amazon more than five years ago, Vass was president and CEO of Liquid Robotics, which designs and builds autonomous robots for the energy, shipping, defense, communications, scientific, intelligence and environmental industries.

How has the vision for AWS storage changed since the object-based Simple Storage Service (S3) launched in 2006?

Bill Vass: Originally, it was very much focused on startups, developers and what we call webscale or web-facing storage. That's what S3 was all about. Then as we grew in the governments and enterprises, we added things like [write once read many] WORM, [recovery point objective] RPO for cross-region replication, lifecycle management, intelligent tiering, deep archive. We were the first to have high-performance, multi-[availability zone] AZ file systems. Block storage has continued to be a mainstay for databases and things like that. We launched the first high-performance file system that will rival anything on prem with FSx for [high-performance computing] HPC. So, we ran Lustre in there. And Lustre gives you microsecond latency, 100 gigabits per thread, connected directly to your CPU.

The other thing we did at Re:Invent [2018] was the FSx for SMB NTFS Windows. At Re:Invent this year, we launched the ability to replicate that to one, two or three AZs. They added a bunch of extra features to it. But, you can expect us with FSx to offer other file systems as well. There's about 35 different file systems that enterprises use. We can support many – really anything with FSx. But we will roll them out in order of priority by customer demand.

What about Amazon Elastic File System?

Vass: Elastic File System, which is our NFS 4 file system, has got single-digit millisecond response. That is actually replicating across three separate buildings with three different segments, striping it multiple times. EFS is an elastic multi-tenant file system. FSx is a single-tenant file system. To get microsecond latency, you have to be right there next to the CPU. You can't have microsecond latency if you're striping across three different buildings and then acknowledging that.

Do you plan to unify file storage? Or, do you plan to offer a myriad of choices?

Vass: Certainly, they're all unified and can interoperate with each other. FSx, S3, intelligent tiering, all that kind of stuff, and EFS all work together. That's already there. However, we don't think file systems are one size fits all. There's 35 different file systems, and the point of FSx is to let people have many choices, just like we have with databases or with CPUs or anything like this. You can't move a load that's running on GPFS into AWS without making changes for it. So you'd want to offer that as a file system. You can't move an HPC load without something like FSx Lustre. You can't move your Windows Home directories into AWS without FSx for Windows. And I would just expect more and more features around EFS, more and more features on S3, more and more features around FSx with more and more options for file systems.

So, you don't envision unifying file storage.

Vass: There will be a central storage management system coming out where you'll see it just like we have a central backup system now. So, they'll be unified at that level. There'll be a time when you'll be able to access things with SMB, NFS and object in the same management console and on the same storage in the future. But that's not really unified, right? Because you still want to have the single-tenant operating environment for your Windows. Microsoft does proprietary extensions on top of SMB, so you'll need to run Windows underneath that. You can run something like [NetApp] OnTap, which also runs on AWS, by the way. And it does a great job of emulating NFS 4, 3, and SMB. But it's never going to be 100% Windows compatible. So for that, you're still going to want to run the Windows-native environment underneath.

I'd love to have one solution that did it all, but when you do that, what you usually end up with is something that does everything, but none of it well. So, you're still going to want to have your high-performance object storage, block storage, elastic file systems and your single-tenant file systems for the foreseeable future. They'll all interoperate with each other. They all get 11 nines durability by snapshotting or direct storing. You're still going to have your archive storage. You don't really want an archive system that operates the same as the file system or an object system.

How will the management console work to manage all the systems?

Vass: Since we unified backups with AWS Backup, you can take a look at that one place where we're backing everything up in AWS. Now, we haven't turned every service on. There's actually 29 stateful stores in AWS. So, what we're doing with backup is adding them one after another until they're all there. You go to one place to back everything up.

We'll add a storage management console. Today, you would go to an S3 console, an FSx console, an EFS console and a relational database console, then an Aurora console, then an EBS console. There'll be one system management console that will let you see everything in one place and one place where you'll be able to manage all of it as well. That's scheduled for some time next year.

I've been hearing from enterprise customers that it can get confusing and overwhelming to keep track of the breadth of AWS storage offerings.

Vass: Let me counter that. We listen to our customers, and I guarantee you at Re:Invent this year, each customer I met with, one of those services that we added was really important to them, because remember, we're moving everything from on prem to the cloud. … There are customers that want NFS 3 still. There's customers that want NFS 4. There's customers that want SMB and NTFS. There's customers that want object storage. There's customers that want block storage. There's customers that want backups. If we did just one, and we told everyone rewrite your apps, it would take forever for people to move.

The best things people can do is get our roadmaps. We disclose our roadmaps under NDA to any customer that asks, and we'll show them what's coming and when it's going to come so that they can have some idea if they're planning and when we're going to solve all of their problems. We've got 2.2 million customers, and all of them need something. And they have quite a variability of needs that we work to meet. So, it's necessary to have that kind of innovation. And of course, we see things our customers do all the time.

So, AWS storage is basically going for the ocean and aiming to get every customer away from a traditional storage vendor.

Vass: I wouldn't say it that way. I'd say we want to enable every customer to be able to use the cloud and Outpost and Snowball and Storage Gateway and all of our products so they can save money, be elastically scaling, have higher durability and better security than they usually do on prem.