DataStax extends Astra Streaming event data platform
The data platform vendor extended its Apache Pulsar-based Astra Streaming platform with support for more streaming protocols, including Kafka, JMS and RabbitMQ.
DataStax became active in the data streaming market after it acquired Kesque in January 2021. The vendor was building a platform based on the open source Apache Pulsar technology.
The Kesque technology became the original foundation for the Astra Streaming service, which provides a commercially supported event data streaming capability. While Astra Streaming started out only supporting Pulsar, with the new release DataStax has extended the technology to support Apache Kafka, open source RabbitMQ and Java Message Service (JMS) data streams as well.
DataStax has multiple competitors in the streaming space, including Confluent, which is one of the leading contributors to Kafka, as well as Pulsar-based vendors, including StreamNative.
While Astra Streaming now has extended its protocol support beyond Pulsar, it still has a few gaps to fill. Among the other popular formats that Astra Streaming doesn't currently support are MQTT and Apache ActiveMQ, which are commonly used for IoT applications.
Among DataStax's Astra Streaming users is Michael Smith, director of engineering at financial services company Commonstock in San Francisco.
We use Astra Streaming as a mechanism to generate events on services that other services can subscribe to.
Michael SmithDirector of engineering, Commonstock
The company's data architecture provides services for financial market data, user portfolio data and user comments, and the services communicate with each other via APIs.
Also, Commonstock has a service that recommends content to display in a user's feed. In order to make recommendations, the platform needs to get metadata from multiple services in real time. That's where Astra Streaming fits into the company's IT architecture.
"We use Astra Streaming as a mechanism to generate events on services that other services can subscribe to," Smith said.
An example is when Commonstock recommends content based on whether someone a customer is following clicks the "like" button for a trade that someone else made, Smith said.
Event data streaming beyond Pulsar with Starlight
The effort to enable the Pulsar-based Astra Streaming service to support other event data streaming technologies comes from a set of open source efforts led by DataStax known as Starlight.
Starlight provides a compatibility layer for different protocols, on top of Pulsar. For Kafka, RabbitMQ and JMS, DataStax developers working on Starlight had to understand how each specific protocol works to support users that want to run those protocols with an Astra Streaming deployment, according to Chris Latimer, vice president of product management at DataStax.
The support for Kafka, RabbitMQ and JMS in Astra Streaming is bidirectional, meaning the service can both import and export event data streams in any of those formats, in addition to Pulsar.
Connecting Astra Streaming to a larger data platform with Astra DB
Streaming data on its own can help organizations with operational dashboards and messaging. Streaming data can also be useful for data analytics and machine learning, but it often needs to be in a database, which is where DataStax's Astra DB cloud service fits in.
In March, DataStax said it was using the Astra Streaming technology to enable change data capture capabilities for Astra DB. While Astra Streaming was not generally available at that time, DataStax was committed to providing full support, but only for the change data capture application with Astra DB, according to Latimer.
Now that Astra Streaming is generally available, Latimer said DataStax is supporting Astra Streaming with Astra DB for any application. For example, data can now be ingested into Astra DB and then used for data analytics.
