Fluent Bit is a lighweight data collector which can be used for log aggregation in microservices, Kuberneted clusters, for basic log analysis, collecting incoming data streams from sensors etc.
Log Analysis is slowly becoming a major area of research and development with distributed services gaining popularity and Kubernetes and Docker leading the way for containerisation, a lot of new services are being developed to solve the problem of collecting data from different sources and storing it to some destination.
Fluent Bit is such a service, which is very easy to install, configure and use.
In this tutorial we will cover how you can easily install Fluent Bit on a Linux machine to start collecting data.
Only Linux operating system's various distributions are supported by Fluent Bit as of the latest 1.3 version of Fluent Bit.
Following are the Linux flavours which are supported:
|CentOS (works with Redhat too)||x86_64|
|Debian 8 (Jessie)||x86_64|
|Debian 9 (Stretch)||x86_64|
|Raspbian 8 (Debian Jessie)||AArch32|
|Raspbian 9 (Debian Stretch)||AArch32|
|Ubuntu 16.04 (Xenial Xerus)||x86_64|
|Ubuntu 18.04 (Bionic Beaver)||x86_64|
While installing Fluent Bit you should not worry about CPU and resource utilization as Fluent Bit doesn't require much resources.
Fluent Bit is distributed with the name td-agent-bit for Linux. Now, let's see how we can install the td-agent-bit service on different Linux distributions, one by one.
To install td-agent-bit on your Ubuntu server you will have to add the Fluent Bit repository to your Ubuntu OS's sources list so that when we run the install command the td-agent-bit package can be easily downloaded from the 3rd party repository.
Also, to access the 3rd party repository we will need to have the public key to download the required packages from it, so let's start by adding the public key for the Fluent Bit repository server. Run the following command to do so,
$ wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
This will add the GPG Key of the Fluent Bit repository server to your keyring.
Then we will add the URL of the Fluent Bit repository to our sources list available at /etc/apt/sources.list. For this you must have root access.
Run the following command to open the file in vim editor, or any editor of your choice,
i to start INERT mode in vim editor. and add the following line at the end:
deb https://packages.fluentbit.io/ubuntu/bionic bionic main
If you have Ubuntu Xenial, add the following line:
deb https://packages.fluentbit.io/ubuntu/bionic bionic main
Once you are done, save the sources list file by pressing ESCAPE and then typing
!wq followed by Enter. This is the command to write and close any opened file in Vim editor.
Now that we have added a new repository, we will run the
apt command to update the apt database.
$ sudo apt-get update
Now we will run the
apt-get command to download and install td-agent-bit service on our Ubuntu machine.
$ sudo apt-get install td-agent-bit
To see the commands to start the service, check status of the service and where the configuration files are located, go to the next heading of this tutorial(skip the Installation in CentOS and Debian).
The installation steps for downloading and installing td-agent-bit on Debian OS is same as that of Ubuntu. In Debian Linux distribution too, we use the
apt-get command to manage packages.
The only difference is that in the sources list, you will have to add the following line at the end:
For Debian 10 (Buster)
deb https://packages.fluentbit.io/debian/buster buster main
For Debian 9 (Stretch)
deb https://packages.fluentbit.io/debian/stretch stretch main
For Debian 8 (Jessie)
deb https://packages.fluentbit.io/debian/jessie jessie main
Rest every step remains the same.
In case of CentOS or Redhat Linux distribution, yum command is used to install any new service on these machines. Fluent Bit also has a Yum repository which we will be adding to our machine's yum repository database.
To do so, create a new file with name td-agent-bit.repo in the directory /etc/yum.repos.d/ using any text editor of your choice. We will be using VIM editor.
Go to the specified directory:
Create and open a file with name td-agent-bit.repo
i to start INERT mode in vim editor. and paste the following text to the file:
[td-agent-bit] name = TD Agent Bit baseurl = https://packages.fluentbit.io/centos/7/$basearch/ gpgcheck=1 gpgkey=https://packages.fluentbit.io/fluentbit.key enabled=1
Once you are done, save the file by pressing ESCAPE and then typing
!wq followed by Enter.
Then run the following yum command to download and install td-agent-bit service on your machine,
yum install td-agent-bit
Once you have successfully installed the td-agent-bit service on your machine, you can run the following command to start it.
$ sudo service td-agent-bit start
This is same for all the Linux distributions mentioned above.
Then to check the status of the td-agent-bit service, you can run the following command.
$ sudo service td-agent-bit status
You should see the output similar to this for the status command:
td-agent-bit.service - TD Agent Bit
Loaded: loaded (/lib/systemd/system/td-agent-bit.service; disabled; vendor preset: enabled)
Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
Main PID: 6739 (td-agent-bit)
??6739 /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
The configuration files for td-agent-bit service can be found in /etc/td-agent-bit/ directory which can be used to set basic configurations like specifying the INPUT plugin, OUTPUT destinations, PARSER, FILTER etc.
You will find the following configuration files there:
parsers.conf - This file has some sample regex PARSERS defined for parsing Apache logs, Apache error logs, Nginx logs etc. We will learn in upcoming tutorials how we can define parser for parsing logs and for other data collection.
td-agent-bit.conf - This configuration file has all the main configurations like SERVICE, INPUT, OUTPUT, FILTER etc.
By default, Fluent Bit is configured to read the CPU metrics and publish them on standard output. In the next tutorial, we will cover how we can setup Fluent Bit, Elasticsearch and Kibana to aggregate logs, store logs in Elasticsearch and visualize them using Kibana.
If you liked this article, do subscribe to our newsletter, you will find the form on the right sidebar.