Installing Fluent Bit on Linux (Ubuntu, Debian and CentOS)

Fluent Bit is a lighweight data collector which can be used for log aggregation in microservices, Kubernetes clusters, basic log analysis, collecting incoming data streams from sensors, etc.

Log Analysis is slowly becoming a major area of research and development with distributed services gaining popularity and Kubernetes and Docker leading the way for containerization, a lot of new services are being developed to solve the problem of collecting data from different sources and storing it to some destination.

Fluent Bit is such a service, which is very easy to install, configure and use.

In this tutorial, we will cover how you can easily install Fluent Bit on a Linux machine to start collecting data.

Fluent Bit Supported Platforms

Only the Linux operating system's various distributions are supported by Fluent Bit as of the latest 1.3 version of Fluent Bit.

Following are the Linux flavors which are supported:

While installing Fluent Bit you should not worry about CPU and resource utilization as Fluent Bit doesn't require much resources.

Download and Install TD Agent Bit

Fluent Bit is distributed with the name td-agent-bit for Linux. Now, let's see how we can install the td-agent-bit service on different Linux distributions, one by one.

Fluent Bit Installation on Ubuntu

To install td-agent-bit on your Ubuntu server you will have to add the Fluent Bit repository to your Ubuntu OS's sources list so that when we run the install command the td-agent-bit package can be easily downloaded from the 3rd party repository.

Also, to access the 3rd party repository we will need to have the public key to download the required packages from it, so let's start by adding the public key for the Fluent Bit repository server. Run the following command to do so,

$ wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

This will add the GPG Key of the Fluent Bit repository server to your keyring.

Then we will add the URL of the Fluent Bit repository to our sources list available at /etc/apt/sources.list. For this, you must have root access.

Run the following command to open the file in Vim editor, or any editor of your choice,

vi /etc/apt/sources.list

Press i to start INERT mode in the Vim editor. and add the following line at the end:

deb https://packages.fluentbit.io/ubuntu/bionic bionic main

If you have Ubuntu Xenial, add the following line:

deb https://packages.fluentbit.io/ubuntu/bionic bionic main

Once you are done, save the sources list file by pressing ESCAPE and then typing !wq followed by Enter. This is the command to write and close any opened file in the Vim editor.

Now that we have added a new repository, we will run the apt command to update the apt database.

$ sudo apt-get update

Now we will run the apt-get command to download and install td-agent-bit service on our Ubuntu machine.

$ sudo apt-get install td-agent-bit

To see the commands to start the service, check the status of the service, and where the configuration files are located, go to the next heading of this tutorial(skip the Installation in CentOS and Debian).

Fluent Bit Installation on Debian

The installation steps for downloading and installing td-agent-bit on Debian OS are the same as that of Ubuntu. In the Debian Linux distribution too, we use the apt-get command to manage packages.

The only difference is that in the sources list, you will have to add the following line at the end:

For Debian 10 (Buster)

deb https://packages.fluentbit.io/debian/buster buster main

For Debian 9 (Stretch)

deb https://packages.fluentbit.io/debian/stretch stretch main

For Debian 8 (Jessie)

deb https://packages.fluentbit.io/debian/jessie jessie main

Rest every step remains the same.

Fluent Bit Installation on CentOS

In the case of CentOS or Redhat Linux distribution, the yum command is used to install any new service on these machines. Fluent Bit also has a Yum repository which we will be adding to our machine's Yum repository database.

To do so, create a new file with the name td-agent-bit.repo in the directory /etc/yum.repos.d/ using any text editor of your choice. We will be using the VIM editor.

Go to the specified directory:

cd /etc/yum.repos.d/

Create and open a file with the name td-agent-bit.repo

vi td-agent-bit.repo

Press i to start INERT mode in vim editor. and paste the following text to the file:

[td-agent-bit]
name = TD Agent Bit
baseurl = https://packages.fluentbit.io/centos/7/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1

Once you are done, save the file by pressing ESCAPE and then typing !wq followed by Enter.

Then run the following yum command to download and install td-agent-bit service on your machine,

yum install td-agent-bit

Start td-agent-bit Service

Once you have successfully installed the td-agent-bit service on your machine, you can run the following command to start it.

$ sudo service td-agent-bit start

This is the same for all the Linux distributions mentioned above.

Then to check the status of the td-agent-bit service, you can run the following command.

$ sudo service td-agent-bit status

You should see an output similar to this for the status command:

td-agent-bit.service - TD Agent Bit
Loaded: loaded (/lib/systemd/system/td-agent-bit.service; disabled; vendor preset: enabled)
Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
Main PID: 6739 (td-agent-bit)
Tasks: 1
Memory: 656.0K
CPU: 1.393s
CGroup: /system.slice/td-agent-bit.service
??6739 /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
...

TD Agent Bit Configuration Files

The configuration files for td-agent-bit service can be found in /etc/td-agent-bit/ directory which can be used to set basic configurations like specifying the INPUT plugin, OUTPUT destinations, PARSER, FILTER etc.

You will find the following configuration files there:

1. parsers.conf - This file has some sample regex PARSERS defined for parsing Apache logs, Apache error logs, Nginx logs etc. We will learn in upcoming tutorials how we can define parser for parsing logs and for other data collection.

2. plugins.conf

3. td-agent-bit.conf - This configuration file has all the main configurations like SERVICE, INPUT, OUTPUT, FILTER, etc.

By default, Fluent Bit is configured to read the CPU metrics and publish them on standard output. In the next tutorial, we will cover how we can set up Fluent Bit, Elasticsearch, and Kibana to aggregate logs, store logs in Elasticsearch, and visualize them using Kibana.

Conclusion

With the increasing complexity of modern IT infrastructures, it has become essential to have a robust log management system in place. Fluent Bit, with its simplicity, flexibility, and powerful capabilities, proves to be an invaluable tool for collecting, processing, and forwarding log data efficiently on Linux distributions like Ubuntu, Debian, and CentOS. By following the comprehensive installation guide provided in this article.

If you liked this article, do subscribe to our newsletter, you will find the form on the right sidebar.

Frequently Asked Questions(FAQs)

1. What is Fluent Bit?

Fluent Bit is a lightweight and efficient log processor and forwarder designed to collect, filter, and transform log data. It offers seamless integration with various data sources, making it an ideal choice for streamlining log management.

2. Why should I use Fluent Bit?

Fluent Bit provides an array of benefits, including high performance, low resource consumption, real-time log processing, and compatibility with multiple data sources. It simplifies log collection and analysis, enabling better troubleshooting, monitoring, and insights.

3. Which Linux distributions are supported by Fluent Bit?

Fluent Bit supports a wide range of Linux distributions, including Ubuntu, Debian, and CentOS, among others. It ensures compatibility across different systems, making it highly versatile for various deployments.

4. Is Fluent Bit difficult to install?

No, Fluent Bit installation is straightforward and well-documented. With the provided step-by-step instructions for Ubuntu, Debian, and CentOS in this article, you can easily set up Fluent Bit on your Linux system.

5. Can I customize Fluent Bit to fit my specific needs?

Absolutely! Fluent Bit offers extensive configuration options, allowing you to tailor its behavior according to your requirements. You can define filters, parsers, and output plugins, ensuring seamless integration with your existing log management infrastructure.

You may also like:

• Installing Kubernetes Helm on Windows

• Installing Fluent Bit on Linux (Ubuntu, Debian and CentOS)

• Setup Fluent Bit with Elasticsearch and Kibana (EFK) for Log Management on Linux Machine (Non Kubernetes)

• Setup Fluent Bit with Elasticsearch Authentication enabled in Kubernetes - Part 3