Meta and partners build Velox open source execution engine
The Facebook parent company, along with multiple contributors, including Ahana, Voltron Data and Intel, are developing a new open source technology to make data processing faster.
Meta, along with a group of its partners, on Wednesday revealed details about its new open source Velox technology that aims to provide a unified execution layer for data management and queries.
Velox is an effort to accelerate data management and data queries by replacing existing execution layers within different technologies, including Apache Spark and Presto. The execution layer is the component that includes the code that handles the movement and processing of data.
While Meta is starting to integrate Velox inside of Meta's platforms, the technology isn't ready for broader production use.
A key goal of the Velox technology is to provide a common standard for execution of data management and processing.
Most data modernization initiatives, including this one, aim for consolidation and simplicity, noted Kevin Petrie, an analyst at Eckerson Research.
But the reality is that most enterprise data environments grow more heterogenous by the day. Multiplying workloads often have specialized requirements, and that complexity extends to data processing engines, Petrie said.
"Enterprises need to simplify how they build, deploy, integrate, reuse and adapt their processing engines," he said. "Velox seeks to help by offering an execution engine that unifies common engine components. This has the potential to make engines more modular, interoperable and reusable, thereby simplifying data environments."
How Velox unifies execution engines for data
Velox is not a tool for data analysts, said Philip Bell, developer advocate at Meta. Rather, Velox is a modular library to be used by those who build and maintain large-scale data processing and storage platforms and is a replacement for the compute layer in those platforms, he said.
Enterprises need to simplify how they build, deploy, integrate, reuse and adapt their processing engines.
Kevin PetrieAnalyst, Eckerson Research
"Velox is a self-contained execution engine that replaces existing layers within data systems," he said.
An execution engine handles some of the computation-intensive work inside of a data computation engine. Velox can replace multiple execution engines found in various data computation engines to streamline data workflows, according to Bell.
Meta uses several different data computation engines for different tasks. Specifically, Bell noted that Meta is looking to replace the execution layer of Presto and Spark with Velox within Meta.
"We are actively integrating Velox into several production systems at Meta," he said. "There are several milestones ahead of us before Velox is production ready more broadly and we hope many will join us in achieving those goals."
Why Ahana is contributing to the Velox unified execution engine
Ahana is among the contributors to the Velox open source effort led by Meta. Steven Mih, co-founder and CEO of Ahana, explained that Velox provides a code acceleration library written in the C++ programming language that can plug into different query engines, including Presto.
Today Presto uses the Java programming language to power the execution layer that communicates with different data sources to process and query data. The goal with Velox's new execution engine is to accelerate the processing with an optimized C++ code base.
While he declined to say when Ahana will put Velox into production code, Mih noted that he expects there to be a positive impact on performance when it's ready. Mih also said he is hopeful the Velox code will move to a third-party open source organization, such as the Linux Foundation or the Apache Software Foundation, at some point in the future.
"I'm a big believer in having third-party governance for open source versus it being controlled by any one company or especially one vendor," Mih said.
