Confluent Platform 6 brings cluster linking to Apache Kafka

Confluent Platform 6 metamorphosis

Confluent Platform 6.0 is the first fruit of Project Metamorphosis, and it is clear that more capabilities are coming, said IDC analyst Stewart Bond.

"Confluent Platform 6.0 is adding features around Kafka that will make it easier for developers to build cloud-native data applications against and increase adoption of real-time stream processing workloads," Bond said. "Business has always been in real time, but technology limitations of the past have constrained it from computing in real time."

Kafka cluster linking

Specifically regarding the new cluster linking capability, Bond noted that at a high level, cluster linking may look like basic federation, but underneath is where it is more than that.

Confluent Platform 6.0 is adding features around Kafka that will make it easier for developers to build cloud-native data applications against and increase adoption of real-time stream processing workloads.

Bond noted that cluster linking preserves offsets across the cluster. In an independent cluster, offsets are used to manage the sequence in which data messages are delivered to a consumer and ensure consumers don't get multiple copies of the same message.

"Offsets are very important in horizontally scaled environments where multiple instances of processes can be running at the same time, reading from the same cluster," Bond said. "With cluster linking, multiple clusters can be federated, data can be shared across clusters, and with offsets being preserved, processes can also be shared across clusters. "

Cluster linking uses the Kafka binary protocol. Addison Huddy, group product manager at Confluent, noted that with cluster linking, Kafka clusters running across different deployments across cloud providers or on premises can be connected together.

"Cluster linking allows software architects to create a global mesh of Kafka," Huddy said.

ksqlDB now production-ready

The ksqlDB technology is a database that is purpose-built for Kafka event streaming. Confluent Platform 6.0 promotes two of ksqlDB's most powerful features, its query layer and connector management, to general availability, said Michael Drogalis, Principal Product Manager at Confluent. The platform update itself is not yet generally available.

Drogalis explained that the query layer provides a low-latency interface for asking point-in-time questions. The connector management capability enables users to source and synchronize data with Confluent's portfolio of more than 100 data source connectors.

"Both of these features have been hardened over the last year through continuous usage, both from our customers and our own cloud operations," Drogalis said. "KsqlDB has seen adoption for building both materialized views and streaming ETL [extract, transform, load] pipelines, both of which are crucial for building modern applications."