Using the RESTful API as an onramp to object storage

Object storage is radically changing the storage landscape. It tackles and solves once thought to be intractable storage problems of non-stop scalability, declining resilience, limited data durability, never-ending tech refresh and runaway costs. Storage pros are excited about the potential, especially for the vast majority of their data that happens to be passive or cold.

Yet perhaps the first question these pros ask is how do they effectively move data into and out of an object storage system? It's a valid question because the native application programming interface for object storage is a RESTful API. RESTful is essentially HTTP puts and gets, or the language of the Internet. That language is native to Web browsers but rarely to most IT applications. This is the first of three tips that provide an answer to this most frequent question. It examines the RESTful API, its variations, standards, de facto standards and what has to be done to use it natively -- and the off-the-shelf software that does just that.

The best way to use an object storage system is via the RESTful API. It's the native interface, has the lowest latencies and fastest response times and provides the most control.

A RESTful API is also known as a RESTful Web service. It's an intuitive, easy-to-use resource-oriented model adopted by most Web 2.0 service providers (such as Google, Facebook, LinkedIn and Yahoo) that exposes their services. It's intuitive because it uses the same API humans use to navigate the Web. RESTful APIs explicitly take advantage of HTTP methodologies defined by the RFC 2616 protocol. It simply uses a "PUT" to change the state of or update a resource, which can be an object, file or block; a "GET" to retrieve a resource; a "POST" to create that resource; and a "DELETE" to remove it. Programming to the RESTful API is not difficult. The biggest issue, as with all programming, is quality assurance and testing.

Regrettably, the RESTful API is not airtight with many areas open to creative use. This has led to a variation among object storage providers. Each has a unique RESTful API. But all is not lost. The market leadership of Amazon Web Services and its Simple Storage Service (S3) has made the S3 RESTful API into a de facto standard. The vast majority of object storage providers provide an S3-compatible RESTful API. This means that an application written to take advantage of the S3 RESTful API should work with most object storage. Key word is should. Testing is required because some S3-compatible implementations are not complete, but rather a smaller subset.

Two other RESTful API standards have emerged. The first is an actual defined standard created and managed by the Storage Networking Industry Association (SNIA) standards body called the Cloud Data Management Interface (CDMI). CDMI is a complete RESTful API that goes beyond an application's ability to create, retrieve, update and delete resources in the object storage. It also enables that application to discover the object storage capabilities, manage the containers in which those resources are placed, as well as the resources themselves. Additionally, it allows metadata to be connected to both the resource containers and contained resources. These enhanced capabilities allow management applications to manage the object storage, its containers, accounts, multi-tenancy, security, access, billing, chargeback and more. CDMI is a very useful comprehensive RESTful API. Unfortunately, only four object storage products (DDN Web Object Scaler, Mezeo Cloud Storage Platform, NetApp StorageGRID and Scality Ring) currently provide an API compatible with and certified for CDMI.

Another new and emerging de facto RESTful API standard is the open source object storage OpenStack Swift RESTful API. Even though OpenStack Swift has limited installations and production deployments, many of the object storage vendors have added the Swift RESTful API. They have done so to make their object storage systems plug-and-play-compatible with OpenStack Nova (the open source cloud compute platform) that has a much greater market presence.

In summary, the RESTful API is the most efficient way to on- and off-ramp to object storage. There are currently three standards to pick from: Amazon S3, SNIA CDMI and OpenStack SWIFT. Modifying an application to use a RESTful API is not science. But that is not always possible for all applications in all IT organizations. The next tip examines software that moves data between NAS or SAN storage to object storage as well as software that makes object storage appear as if it were NAS or SAN storage.