How to prepare for the age of edge storage
The extreme storage demands of data-intensive applications are fueling tremendous growth in edge networks, and it’s opening a Pandora’s box of potential problems. While it’s a given that networks and applications such as IoT, connected vehicles, augmented reality, gaming and 5G networks are storage hogs, they also require high data processing requirements to be effective.
International Data Corporation’s “Data Age 2025” report predicts IoT devices alone will create 90 zettabytes of data. That’s 90 sextillion bytes. This data deluge will send storage demands at the edge soaring. According to industry analyst firm Gartner, 75% of the data will be processed at the edge by 2020. That’s a huge shift from today, when researchers say 91% of the data is generated and processed at the central datacenter or cloud.
As network edge processing grows, new technologies are emerging to help enterprises keep pace with rapidly rising storage needs and meet application requirements like lower latencies, reduced cloud egress bandwidths, government regulations and sensitive data security by keeping it local. For example, IoT devices may need to make local processing decisions quickly and not tolerate the latency of reading and writing the data.
The small scale of each edge cloud makes it increasingly difficult to gain economies of scale for the infrastructure. There is less flexibility in resource allocation since there are fewer resources to allocate. All of this makes it even more critical than in core clouds to use the infrastructure efficiently and consolidate resources to fit many diverse workloads on the same infrastructure. The criticality is not only from a total cost of ownership (TCO) consideration, but also to be able to run applications at the edge that need to. After all, it may not be possible to easily add more infrastructure in the same location.
The physical location of edge clouds is based on the proximity to the edge devices it serves. This factor implies special hardware requirements from the infrastructure equipment and might require flexibility in form factors, power, cooling and service mechanical properties. For example, how do technicians access the system to replace a drive? The constraints imposed by the range of physical locations of edge clouds requires a flexible infrastructure that can work with different hardware form factors. Hardware suitable for one edge cloud is not always ideal for or available to another.
Finding the best TCO storage solution for edge clouds
There are several critical factors when considering the right storage solution for edge clouds, but none is more important than TCO. Delivering the optimum TCO typically involves the following:
- Using standard hardware for compute, networking and storage with a low number of server models.
- Deploying low-grade flash drives rather than hard disk drives (HDD), such as quad-level cells and datacenter or even consumer-grade drives.
- Disaggregating storage to allow independent scaling of compute and storage, with more optimized storage use.
- Reducing system service requirements and increasing operational efficiency.
Other factors to consider for edge cloud storage
Edge locations are not well known for being efficiently serviced by technicians since each site is relatively small, likely remote and geographically distributed. This reality makes on-site service costly, so any edge cloud storage solution deployed should help reduce service time. Moreover, when introducing capacity into the edge locations, avoiding drive failure is critical for reducing in-person maintenance requirements. This target can be achieved by moving from HDDs to solid-state drives (SSDs) and having a smart layer that manages the SSDs and reduces their failure rate.
The flash management layer is important for serviceability as many of the drive failures, from an application perspective, are transient failures and are not included in the mean time between failurescalculation. In reality, with a direct-attached storage (DAS) architecture, such transient failures might cause application failure and might even be followed by reconstruction of the data that imposes an unnecessary load on compute and network at the edge. The serviceability issue also comes into play when organizations need to add storage or compute, which must be done in an easy plug-and-play fashion.
With limited edge location space, achieving high density and form factor flexibility in choice and variety is critical. Each edge instance can have different physical requirements while providing the same user experience. One way of achieving high density with limited floor space is by deploying high capacity flash drives and new form factors and implementing an architecture that maximizes resource use and allows the best compute-to-storage ratios.
Another important consideration is that edge locations require special care for security and privacy. In some cases, data privacy is a significant driver for keeping the data at the edge rather than uploading it to a public cloud. For example, as IoT devices are spread throughout the globe, sending the data from these devices to a secure edge cloud is essential. Some security concerns can be addressed with data-at-rest encryption if the drives support it or with a software implementation of data encryption. Other concerns arise when different applications run on the same hardware. Any edge storage solution must provide data and performance isolation guarantees between colocated applications. Each application must only access the data it has permission to access.
NVMe/TCP for edge storage
One of the most practical methods for separating storage from compute is with non-volatile memory express (NVMe)/TCP. With the NVMe/TCP standard ratified in late 2018, production-grade storage solutions for NVMe/TCP are now being built that provide NVMe performance based on a choice of networks without any constraints. NVME/TCP solutions are especially important for edge deployments, where adding network constraints can be impossible. Thanks to this new protocol, new storage solutions are helping organizations disaggregate storage over any IP network, and cluster several proximate edge locations into a high availability storage pool accessible within this cluster or have stateless edge instances seamlessly using storage at the aggregation layer.
These solutions use off-the-shelf, standard servers and allow organizations to choose the servers that best fit the physical edge location, including non-standard form factors. While continuing to leverage the existing supply chain to reduce the overall TCO, the disaggregation of storage from compute enables stateless application servers and allows efficient, quick and independent scale of compute and storage according to application needs. The disaggregation also improves TCO by eliminating the stranded capacity problem present in DAS deployments.
The disaggregation of storage also helps alleviate on-site technician service costs that are often a pain point of edge deployments. Some of the software-defined storage on hyper-converged infrastructure solutions base their handling of drive failures on large-scale deployments and massive numbers of servers and drives participating in the rebuilding process. At the edge, this is not enough since there are fewer infrastructure racks. New NVME/TCP solutions are designed to extend the usage and improve performance of flash memory by taking responsibility for it and generating a precise workload.
When building edge infrastructure for edge cloud computing to process all of the data from billions of IoT devices, storage is an essential building block. Choosing the best storage solution for edge cloud necessitates meeting edge-specific requirements. As organizations look to move more applications to the edge, storage must keep up delivering low latency, low TCO, higher density and smaller form factors. NVME/TCP storage solutions are designed for edge cloud infrastructure and optimally serve the needs of the edge and enable organizations to fully benefit from the potential of the edge infrastructure.
All IoT Agenda network contributors are responsible for the content and accuracy of their posts. Opinions are of the writers and do not necessarily convey the thoughts of IoT Agenda.