How to use Slack and Fluentd to demonstrate social output

Slack has become a leading team messaging collaboration tool with a strong API layer. It includes a free version that is simply limited by the size of conversation archives. As a cloud service as well, it makes for a great tool to illustrate the intersection of Fluentd and actionable log events through social platforms. While the following steps are specific to Slack, the principles involved here are the same for Microsoft Teams, Jabber and many other collaboration services.

If you are already using Slack, it is tempting to use an existing group if you want to run the example. However, you risk irritating other users by filling your channels with test messages. If you aren't using Slack, that's not a problem, a Slack account is easy to configure and use. During the configuration of Slack, ensure you keep a note of the API token -- recognizable by having a prefix of xoxb.

Slack provides a rich set of configuration options for the interaction. Unlike the MongoDB and many IaaS and PaaS-level plugins, Slack is a SaaS service, so the resolution of the Slack instance is both simplified and hidden from us by using a single token (no need for server addresses, etc.). The username isn't about credentials, rather how to represent the bot that the Fluentd plugin behaves as, therefore using a meaningful name is worthwhile. The channel relates to the Slack channel in which the messages sent will be displayed. The general channel exists by default, but if you want to control who sees the messages, then you may wish to create a custom channel in Slack and restrict access to that channel; otherwise everyone within your enterprise Slack setup will see every operational message.

Click the image to learn
more about the book.
Take 35% off any format
by entering ttwilkins into
the discount code box
at checkout.

The message and message_keys attributes work together, with the message using %s to indicate where the values of the identified payload elements are inserted. The references in the message relate to the names of the JSON payload elements listed in the message_keys in order sequence.

The title and title_keys work in the same manner as the message and message_keys but for the title displayed in the Slack UI with that message. In our case, we're just going to use the tag. The final part is the flush attribute --  this tells the plugin how quickly to push the Slack messages to the user. Multiple messages can be grouped if the period is too long. To keep things moving quickly, let's flush every second.

Edit the existing configuration provided (Chapter4/Fluentd/rotating-file-read-slack-out.conf) to incorporate the details captured, in the Slack setup. This is illustrated in Listing 1.

Listing 1. /Chapter4/Fluentd/rotating-file-read-slack-out.conf – match configuration

<match *>
   @type slack
   token xoxb-9999999999999-999999999999-XXXXXXXXXXXXXXXXXXXXXXXX    #A
   username UnifiedFluent    #B
   icon_emoji :ghost:    #C
   channel general    #D
   message Tell me if you've heard this before - %s    #E #G
   message_keys msg    #F #G
   title %s    #H #I
   title_keys tag    #I
   flush_interval 1s    #J
</match>

#A: This is where to token obtained from Slack is put, to correctly identify the workspace and legitimize the connection.
#B: Username that will be shown in the Slack conversation.
#C: Define an individual emoji (icon) to associate the bot.
#D: Which channel to place the messages into, based on the name. In our demo, we're just using the default channel.
#E: Message to be displayed, referencing values in the order provided in the config.
#F: Name the log event's JSON elements to be included in the log event.
#G: These attributes work as a pair.
#H: Title for the message takes the values from title_keys. Using order to map between the config values.
#I: Definition of the message's title.
#J: Defines the frequency at which Fluentd will get Slack to publish the log events.

Before we start running the solution, we also need to install the Slack plugin with the command fluent-gem install fluent-plugin-slack. Once installed, we can start up the log simulator and Fluentd with the following commands:

• fluentd -c Chapter4/Fluentd/rotating-file-read-slack-out.conf
• groovy LogSimulator.groovy Chapter4/SimulatorConfig/social-logs.properties ./TestData/small-source.txt

Once started, if you open the #general channel in the web client or app, you will see messages from Fluentd flowing through.

All the details for the Slack plugin can be obtained from Github. Our illustration of Slack use is relatively straightforward. Through the use of several plugins, we could very easily go from the source tag to addressing the Slack messages to the most relevant individual or group directly, or having different channels for each application and directing the messages to those channels.