Speedb data engine brings key value store to open source
The startup looks to provide a competitive alternative to RocksDB by building out its data technology and community of developers with a move to an open source methodology.
Speedb said it is moving to an open source model for its data engine technology.
The 2020 startup raised $4 million in a seed round of funding in December 2021 to develop its key value store technology aimed at being a drop-in replacement for the open source RocksDB project.
RocksDB is among the most widely deployed key value store technologies and was developed by Meta (formerly Facebook) in 2012. Speedb's goal is to provide easier scalability and better performance than RocksDB. It's a goal the vendor had been pursuing with a closed source technology.
Speedb is now changing its direction and is beginning to embrace an open source model, though the vendor will continue to offer a closed source enterprise edition with additional performance features.
Speedb competes with a number of other vendors as well, including database vendor CockroachDB, which has created its own RocksDB replacement called Pebble. As RocksDB has been in development for a decade, alternative key value store technologies all face the challenge of convincing users that choosing an alternative offers tangible advantages.
One organization that made the shift from RocksDB to Speedb is security vendor XM Cyber.
The Flink clusters deployed by XM Cyber had been using RocksDB as the key value store for storing data. But there were performance problems that impeded scalability, said Yaron Shani, tech lead and senior cyber security researcher at XM Cyber.
Shani and his team wrote a benchmark to measure the impact of the RocksDB performance problem. Shani said that when the same benchmark was run with Speedb, the performance was considerably better. He also noted that the support from Speedb as a vendor was helpful as well.
While RocksDB works adequately for many organizations, it is a purely open source project and doesn't have the support options that a commercial vendor like Speedb offers. With an open source project, it's up to the users to fine-tune and optimize performance.
For Shani, Speedb's move to open source won't affect XM Cyber.
"We have a commercial contract with Speedb and whether the code is open source or not is less crucial for us," Shani said. "Usually developer tools are much better being open source and can greatly help adoption, so I wish best of luck to Speedb."
Why Speedb data engine is going open source
The move to open source, revealed by Speedb on Nov. 9, doesn't actually mark the first time the company has had open source code.
Usually developer tools are much better being open source and can greatly help adoption, so I wish best of luck to Speedb.
Yaron ShaniTech lead, senior cyber security researcher, XM Cyber
Guy Meir, CTO of Speedb, said that the vendor has had code on the Github code repository for several months as an initial attempt to attract developers, not as an effort to build an open source project where there is a community of contributors.
"We started with a relatively cynical view. We figured out that we need to get the developers on board, so we should give them an open source version," Meir said.
That initial approach has now changed. He and his colleagues have come to realize the benefit and opportunity of building software in an open source approach, Meir said. The open source approach to development includes both a way of building software collaboratively and the benefits of building a community of actively engaged developers, he said.
Enterprise vs. open source features in Speedb data engine
Though Speedb has now embraced the open source model for building technology, the vendor said it will continue to develop its enterprise edition as well. The enterprise version of Speedb provides commercial support as well as high-performance features that are not available in the open source version.
Meir emphasized that it's still early in the process of Speedb moving to the open source model, though there are already several powerful features in the open source release today. For example, the open source version includes a new bloom filter methodology that Speedb developed to improve performance.
Meanwhile, an enterprise capability that is not now available in the open source edition, but might be in the future, is compaction.
Both RocksDB and Speedb use what is known as a log-structured merge-tree (LSM) data structure. In the Speedb enterprise edition, the compaction feature sorts and merges data for the LSM in an optimized approach that provides a performance boost.
Looking forward, Meir said that more features will come to the open source version in the coming months, including improved memory management and a performance analyzer.
