Confluent brings designer and governance to streaming data

The essential nature of streaming data for enterprises

"Streaming data is becoming more and more common in organizations' information infrastructures," said Dave Menninger, analyst at Ventana Research. "It's becoming a competitive necessity for organizations to provide real-time responsiveness to their customers, prospects, employees and partners."

Streaming data is becoming more and more common in organizations' information infrastructures. It's becoming a competitive necessity for organizations to provide real-time responsiveness to their customers, prospects, employees and partners.

Looking specifically at the new capabilities Confluent introduced on Tuesday, Menninger said the vendor is following a well-established pattern. As new data technologies become widely available, they need to be easier to use, and they need to be governed. Streaming data is no different.

"These features also have the potential to bring a new audience to Confluent's streaming data capabilities as well," Menninger said. "No longer will you have to know all the nuances of Kafka in order to easily take advantage of them."

Building streaming data pipelines with Confluent's new designer

Confluent developed the stream designer to enable organizations to better organize and build streaming data pipelines, said Chad Verbowski, senior vice president of engineering at Confluent.

Although Kafka is widely deployed, it can be challenging for organizations to effectively use data coming from multiple Kafka streaming data sources. A way to deal with that challenge is with a streaming data pipeline, which is what the new stream designer enables users to build.

For example, for a user transaction, one stream might include the purchase information while another provides the user location. An organization needs to be able to combine information from both of those streams to make a decision on whether to approve the transaction. With stream designer, the feature creates a SQL query that can analyze multiple Kafka streams simultaneously to generate a result.

Stream designer provides a visual interface that enables users to see the different Kafka topic streams and build the queries that will create real-time streaming data pipelines. The stream designer feature also has a pre-built connector in Confluent Cloud that enables users to connect to data sources from cloud applications such as Salesforce, Workday or SAP to develop the streaming data pipeline.

Managing data streams with advanced stream governance

The new advanced stream governance capability in Confluent Cloud provides organizations with security controls for how streaming data can be shared and used.

The stream governance feature works with an enhanced streaming data catalog capability in Confluent Cloud to capture metadata about a stream, including lineage. With the streaming data catalog, organizations can better identify what is in a stream, and define how it can be accessed and shared both within the organization as well as externally with business partners.

"One of the big things that we're doing with advanced data governance is allowing more business metadata in the streaming catalog," Verbowski said. "We think that people will use that to very quickly identify which topics they need to look at, and also be able to understand more contextually what they should be expecting in those streams."