Alternatives to RESTful API for accessing object storage

The RESTful API is the native access to object storage and the most efficient way to on- and off-ramp object storage. It provides the lowest latency and the highest level of management control. There are three current de facto and published standards: Amazon Simple Storage Services, SNIA CDMI and OpenStack Swift, as well as the proprietary variations, which I have detailed in a previous tip on the topic. However, there are times when it's not practical -- or possible -- to modify an application. This tip examines the alternative software options to modifying an application using object storage RESTful API.

There are many applications and file systems that just cannot be modified to utilize a RESTful API because of license reasons, warranties, dearth of programming skills, etc. For those applications to take advantage of object storage without modification means the object storage has to look like traditional target storage to the application file system. In other words, the object storage has to look like a mountable POSIX-compliant file system such as NFS, CIFS and/or the Hadoop Distributed File System (HDFS), or a SAN storage target via the iSCSI protocol. Fortunately, there is a considerable amount of software available that can do that today.

The most common is conversion, or gateway, software that's packaged as a feature of the object storage software. In this manner, the object storage looks and feels like a mountable POSIX-compliant file system or an iSCSI (block) target storage system. There are upsides and downsides to this approach. The advantages come from the object storage system responding or acting the same as a traditional storage system. No changes or alterations are required to the application or file system. It also allows files stored via NFS to be accessed via other file protocols such as CIFS or HDFS and vice versa. In addition, it allows objects stored via the RESTful API to be accessed as files and files to be accessed as objects.

There are downsides or tradeoffs with this approach as well. First and foremost, there is greater latency, leading to longer response times. Anytime one interface must be converted to another, there will be more latency. In addition, the NAS experience is not identical. This is because the software conversion function is an interface convenience, not a replica of a NAS system with all of its features. The iSCSI response times also tend to be slower than native iSCSI storage (latency again) and are not accessible any other way than as iSCSI blocks.

Many of these conversion features (in some cases, all) are available on object storage from Amplidata Amplistor, Caringo CAStor, CEPH open source, Coho DataStream, EMC Atmos, NetApp StorageGRID, Quantum Lattus and Scality RING.

A significant variation of the bundled gateway features comes from Cleversafe dsNet. Cleversafe delivers Hadoop integration as an optional bundle with its dsNet object storage. Rather than providing a HDFS conversion feature, Cleversafe integrated Hadoop to work natively with its RESTful object storage API, eliminating the additional latency.

Another significant variant is from Joyent Manta. Joyent delivers combined compute and object storage in each node. Applications developed and running on a Manta node automatically take advantage of the Joyent object storage. This is similar to the way Amazon Web Services (AWS) automatically takes advantage of Amazon Simple Storage Service in the Amazon cloud.