Storage and database innovation evolve in tandem

Corresponding evolution

Storage and database systems have been on the same technology and deployment methodology curves, said Ken Claffey, vice president and general manager of enterprise data solutions at storage products provider Seagate Technology. Over time, SQL databases have evolved from vertically scalable systems to NoSQL databases, which are horizontally scalable distributed systems. Similarly, storage technologies have evolved from vertically scaled arrays to horizontally scaled-out distributed storage systems. "Both [technologies] have also undergone deployment methodology changes, from on-premises proprietary systems to software-defined open source cloud storage models," Claffey said.

The NoSQL revolution, which eliminated the need to create a complex, difficult-to-manage data model simply for the purpose of reducing data duplication, correlates closely with the arrival of PCIe-based SSDs in the 2007-2008 timeframe, said Glauber Costa, distinguished engineer at database developer ScyllaDB. The subsequent arrival of NVMe technology has helped to further accelerate database performance, he added.

Looking forward, persistent memory promises a new generation of fast and powerful databases, capable of handling video, IoT and other complex data. "With persistent memory, you're going to see everyone trying to push from single millisecond (ms) latencies toward dozens of microseconds (µs)," Costa said. He also noted that storage has moved from a world where 10 ms access was acceptable on HDDs, to one where it's possible to obtain 400 µs performance on SATA SSDs, to 10-100 µs on NVMe devices. "Next, we're talking 100-200 nanoseconds -- RAM-level -- on persistent memory," he said.

Storage providers are also recognizing the need to enhance their offerings in ways beyond sheer speed to meet the demands of larger, more complicated and increasingly vulnerable databases. "We're not just talking about parallelism for throughput, latency or density," Costa said. "For example, with increasing requirements for privacy and security in databases, many NVMe providers now include data-at-rest encryption."

Speed and scalability

Next-generation databases require faster storage and highly scalable architectures. "Today's NVMe controllers can handle four times more parallel I/O commands than SAS [and] SATA SSD controllers, with one NVMe drive often capable of providing the performance of an entire all-flash array," said Kirill Shoikhet, chief architect at Excelero, a provider of software-defined block storage for cloud and enterprise applications.

Transactional processing is all about latency and parallelism, and NVMe technology can significantly lower latency, Shoikhet said. "The ability to access NVMe drives remotely in an efficient manner using NVMe-oF opens up many possibilities for running database instances in a data center for a performant database as a service," he noted. "However, the bottlenecks imposed by traditional controller-based architectures implicitly mean they cannot scale performance to allow many simultaneous accesses with low latency. In contrast, approaches based on direct access using NVMe-oF, or similar protocols over [remote direct access memory] networks, enable shared, low-latency access that allows databases to scale to cloud-like environments and retain strong ROI."

What's next for storage and database technology

Storage prices continue to drop rapidly as data demand and use soar. However, despite recent advancements, the scalability and performance of an application can still be limited by the performance and feature set of storage, said Matt Yonkovit, chief experience officer at Percona, a database software and services provider. "As the demand for applications to store more data increases," he added, "so will the demand for faster, more reliable storage."

There are many exciting developments at the intersection of storage and databases, including computational storage, persistent memory and GPU-mapped SQL. Change and innovation in storage and database technology is happening at a rapid pace and is definitely worth monitoring.