The new Native Collectors service in the cloud is an attempt by the time series database vendor to let users connect to message broker services and event streaming data.
Time series database vendor InfluxData added new capabilities to its InfluxDB Cloud service that aim to make it easier to integrate event streaming data.
In February, the vendor added new capabilities to better support IoT and industrial sensors. Those capabilities are being further extended with the InfluxDB Native Connectors service in the cloud to make ingesting and loading data coming from event streaming data and message broker services easier.
The first service supported by InfluxDB Native Connectors is the MQ Telemetry Transport (MQTT) protocol. InfluxData plans on supporting Apache Kafka, AMQP (Advance Message Queuing Protocol) and Amazon Kinesis with its native connector approach in the coming months.
InfluxData competes against a number of database vendors that provide time series capabilities, including MongoDB, which has been building out its time series features in recent releases.
InfluxData is attempting to help end users streamline the process of getting data into InfluxDB Cloud with the introduction of InfluxDB Native Connectors, said Rachel Stephens, an analyst at RedMonk.
A common architectural pattern is to send time series data from an IoT device to an MQTT message broker, which publishes data to a Telegraf collection agent that then passes the data to InfluxDB. The new connectors from InfluxData simplify that approach.
"The Native MQTT collector will enable users to connect their MQTT broker directly to InfluxDB," Stephens said. "The introduction of native connectors should help reduce the level of overhead required to write and maintain MQTT data collection pipelines."
The Native MQTT collector will enable users to connect their MQTT broker directly to InfluxDB. The introduction of native connectors should help reduce the level of overhead required to write and maintain MQTT data collection pipelines.
Rachel StephensAnalyst, RedMonk
Bringing event streaming data to the InfluxDB time series database
Until now, users could add data to the InfluxDB Cloud either by writing code to connect sources or by deploying an agent, such as Telegraf, said Brian Gilmore, director of IoT and emerging technology at InfluxData.
"There really wasn't a way for user to simply point at a data source and just say, 'Collect data from here,'" he said. "What we've done is basically taken up to potentially thousands of lines of code that somebody would be managing externally to do data ingest at large scale, and we just brought it all within the database."
InfluxDB Native Connectors also include a built-in parser that lets users obtain the specific information they need from a data stream.
For example, an MQTT data stream from a sensor could include all kinds of data about an operational environment that isn't all needed in the database. With the connectors, it is now possible to just pull out and ingest data that a developer is looking for -- for example, the temperature in an operational environment such as a warehouse.
Gilmore noted that InfluxData is providing the service via a web-based user interface in the InfluxDB Cloud to make connecting to data sources easier.
How InfluxDB Native Collectors works
In building out the new collector technology, InfluxData has directly integrated a message broker client into the InfluxDB Cloud.
The vendor is using the open source Apache NiFi technology as a foundation for directly ingesting event data streams.
NiFi enables the routing and parsing of messages needed by InfluxDB Native Connectors. With the InfluxDB Cloud, the vendor can turn message and data stream collection into an elastic multi-tenant service, Gilmore said.
"We take full responsibility in the cloud, through that open source software base, to manage all the client data collection," he said.
