AWS Data Exchange and the third-party cloud data marketplace

Among the many data services AWS provides on its cloud platform is the AWS Data Exchange, which the tech giant launched in November 2019.

Over the past year, the service has grown in terms of data feeds and capabilities and many have used it to deal with coronavirus pandemic-related issues.

The exchange enables third-party publishers of different types of data categories spanning financial services, retail, healthcare and life sciences, to provide data feeds that users can subscribe to. Those data feeds can then be incorporated into other services such as data analytics and business to data-driven decision-making.

Leading the AWS Data Exchange is general manager Stephen Orban, a veteran of the data publishing industry. Before AWS, Orban spent three years as the CIO of Dow Jones and before that he was at Bloomberg for 11 years.

In this Q&A, Orban explains why the AWS Data Exchange was created and outlines the challenges and opportunities for third-party data sources.

Why did AWS create the AWS Data Exchange in the first place?

Stephen Orban: One of the things that I kept hearing from customers when I first joined AWS was that it was great that they can move much faster with applications, analytics, and now machine learning models in the cloud, but not a whole lot has changed with respect to how organizations exchange data with one another.

Stephen Orban

We had a number of customers across multiple different industries like financial services, healthcare and life sciences, media and entertainment, even in the public sector, tell us that it's pretty strange that they're still dealing with shipped physical media. We have customers that are still literally shipping hard drives to each other, managing FTP sites and the credentials that have to go on with that format.

So that was the motivation and the reason we built AWS Data Exchange. The goal is to make it easy for AWS customers to find, subscribe to and use third-party data in the cloud, while at the same time make it easy for qualified data providers to reach the millions of AWS customers who are migrating to AWS without them having to build or maintain any infrastructure around data storage, delivery, billing or entitlement.

What are the key challenges for enabling data exchange?

Orban: We hear about a few points of friction both from data providers and data subscribers. It is worth pointing out that particularly in the enterprise space, a number of the data providers we work with are also data subscribers in different parts of their business.

A lot of customers have told us that they spend a bunch of time trying to look for new sources of data, when they'd rather have one place they can go to find it. The second thing we hear from those customers is even if they do find the data they need, they then have to build and maintain a whole bunch of technology infrastructure to get it into their own environments, where they can use it. The problem is that every data provider delivers data a little bit differently.

The goal is to make it easy for AWS customers to find, subscribe to and use third-party data in the cloud.

I worked with an insurance company and they told me they have a 25-person team and the only thing that team does is deal with all the nuances of how all their third-party data vendors deliver the data, because everybody does it a little bit differently.

The first thing we hear, particularly from established data providers, is that they want to be able to reach what they'll often call the long tail of customers that they might not reach today.

Now, a lot of these data providers already have lots of customers, hundreds or even thousands of them, and they tend to be kind of all within the same industry. But they also know that there's millions of customers who are using third-party data in their data lakes, analytics and machine learning applications that they're building and running on AWS. They'd like to have a self-service way to reach all those customers, without having to invest in large sales teams, or digital distribution channels to reach them.

How have organizations used the data exchange during the pandemic?

Orban: There are a lot of data sources that have helped different groups during the pandemic. One example is a large consumer packaged goods customer that distributes a lot of food and goods to both supermarkets and restaurants directly.

When the pandemic hit, people started going out less to eat at restaurants, and therefore more people were going shopping in the supermarkets for their foods. But how people reacted in various parts of the country was very different. Some areas adhere to social distancing and are not going out, much more than others. So, it was very difficult for the company to come up with a one-size-fits-all strategy map. They used a lot of location data, and COVID-19 case data in various locations to determine how to adjust shifts to their supply chain, at the right level in the right places.

What's next for the AWS Data Exchange?

Orban: We're just starting to see the innovation that our customers are doing and we're really excited to continue to support them. Like all of our services, 90% of my roadmap is driven by exactly what our customers tell us they want.

What they tell us they want, maybe not surprisingly, is more data sources, and more diverse kinds of data. So, we're relentlessly working on getting basically any data source that our customers want by working with our existing providers and recruiting new ones. You can expect to see more and more data sources.

This interview was edited for clarity and conciseness.