What you should know about using flash to complement DRAM

Flash has dramatically changed the way storage systems are designed and configured.

Adding flash to a storage system or making a storage system completely out of flash leads to denser, more scalable environments. This drives down the total cost of deployment.

Today, flash is taking on another new role as a complement to dynamic random access memory (DRAM). In this role, flash has the net effect of slow DRAM rather than fast storage.

Why use flash to reduce DRAM needs?

There are three primary reasons for using flash to reduce the amount of DRAM. The first is probably the most obvious: price. Flash is significantly cheaper than DRAM and will likely maintain its price advantage for the foreseeable future. To some extent, it is a case of supply and demand. Flash is being used in almost everything today, from smartphones to enterprise storage systems. There is also significant progress being made to drive down the price of flash, including triple level cell (TLC) and 3-D NAND technology, both of which will likely break into the enterprise market in 2015.

The second reason to consider using flash as memory is its density. There is a limited amount of space in a server, and the more memory capacity you can fit in that space, the better. There are applications that can benefit from all of this capacity, namely virtualization (desktop and server), in-memory databases and high-performance computing (HPC) environments.

The third reason is flash's nonvolatile nature. Unlike RAM, it does not need to be constantly powered. If more of an application's temporal data is stored on flash, recovery should be significantly faster. Think of a server environment that could "go to sleep" like your laptop can. This is beneficial for rapidly recovering from an outage and also for power efficiency.

Flash allows for tiering in server memory just as there is in storage. Not all applications can take advantage of DRAM speeds, so serving data from flash is perfectly acceptable.

Overcoming flash performance problems

Performance is the greatest challenge when using flash to release a server's DRAM requirement. However, there are two areas where its performance can be improved.

The first is in the performance of the flash controller and the way it manages flash. This includes better organization of writes, faster and more efficient error correction, and better wear-leveling techniques.

The second and potentially greater opportunity for performance improvement is the storage interconnect for the flash system. A traditional SCSI type of access, for example, will add a noticeable amount of latency.

Fast flash access

Using flash as a complement to DRAM first became a reality when PCIe-based flash devices that did not need to use the SCSI protocol to be accessed came to market. While this meant loading a driver, it provided the CPU with native access to the flash storage area, resulting in much lower latency than SCSI access.

PCIe-based flash has weaknesses, though; the primary being that the PCIe bus is an architecture that's shared with all sorts of other devices, including network cards and storage controllers. Most servers use multiple PCIe cards at the same time, so a fair amount of bus contention is likely.  Also, many smaller systems, especially blade servers, have a limited number of PCIe slots.

So, PCIe flash can only be used as very fast storage, not as actual RAM. As a result, it is primarily used as a very high-speed virtual memory swap space and high-speed application cache area. For flash to work as a complement to DRAM, systems need a more consistent way to access it.

Flash DIMMs

Consistent, low-latency access is available today in the form of flash dual in-line memory modules (DIMMs).

In this implementation, NAND flash is integrated into a DIMM module and is installed in the DDR memory slots on the server's motherboard. The memory bus is a private network of sorts dedicated solely to memory, so latency is nearly eliminated. The flash DIMM technology enables very high density -- currently 500 GB+ per DIMM slot -- for about the same price as PCIe SSDs.

The downside to flash DIMMs is that they require a motherboard update in order to be supported. Some server manufacturers have performed these upgrades, and it is expected that more will follow suit in 2015. With these updates, flash DIMM can either act as slow RAM, or with an additional driver, can perform as high-performance storage.

Use cases for flash in complement to DRAM

PCIe flash reduces latency when compared with SCSI-based flash devices, which should be particularly noticeable in a performance-sensitive operation like moving data to and from DRAM. However, there is overhead to this paging of memory, which could impact application performance.

Flash DIMMs, when used in complement to DRAM, act as RAM, but slower. Not only do they benefit from an improved bus when compared with PCIe, but memory paging is also less of an issue. The technology continues to improve and can handle a mixed state (DRAM and flash as memory) more intelligently. There is also a continuing improvement in flash DIMM capacity. In 2015, 1 TB DIMMs should be widely available.

No matter the data center size, flash can and should be adopted inside of servers, in addition to playing a role for shared storage. The in-server use case can be as simple as enabling faster boot times, but can also provide very rapid page swaps. The number of page swaps that occur per server should be the gauge as to the value of PCIe flash vs. SCSI-based flash.

Flash DIMMs are for environments that need very high memory concentrations per server. They are also important in environments that need low latency of access, where page swaps and/or access across the PCIe bus can make a difference. These are typically in-memory databases, HPC environments and potentially dense virtual desktop architectures.

About the author:
George Crump is president of Storage Switzerland, an IT analyst firm focused on storage and virtualization.