New Tutorials:   NUMPY    TKINTER    KOTLIN    JAVASCRIPT    SASS/SCSS    PL/SQL    Matplotlib    C++ Programs
CLOSE
   FluentBit  Installation Guide  HowTo  
   Technology    Programming

Installing Fluent Bit on Linux (Ubuntu, Debian and CentOS)

         
  APRIL 15, 2020   by iamabhishek

Fluent Bit is a lighweight data collector which can be used for log aggregation in microservices, Kuberneted clusters, for basic log analysis, collecting incoming data streams from sensors etc.

Log Analysis is slowly becoming a major area of research and development with distributed services gaining popularity and Kubernetes and Docker leading the way for containerisation, a lot of new services are being developed to solve the problem of collecting data from different sources and storing it to some destination.

Installing Fluent Bit on Linux machine

Fluent Bit is such a service, which is very easy to install, configure and use.

In this tutorial we will cover how you can easily install Fluent Bit on a Linux machine to start collecting data.

Fluent Bit Supported Platforms

Only Linux operating system's various distributions are supported by Fluent Bit as of the latest 1.3 version of Fluent Bit.

Following are the Linux flavours which are supported:

Linux Distrbution Architecture
CentOS (works with Redhat too) x86_64
Debian 8 (Jessie) x86_64
Debian 9 (Stretch) x86_64
Raspbian 8 (Debian Jessie) AArch32
Raspbian 9 (Debian Stretch) AArch32
Ubuntu 16.04 (Xenial Xerus) x86_64
Ubuntu 18.04 (Bionic Beaver) x86_64

While installing Fluent Bit you should not worry about CPU and resource utilization as Fluent Bit doesn't require much resources.

Download and Install TD Agent Bit

Fluent Bit is distributed with the name td-agent-bit for Linux. Now, let's see how we can install the td-agent-bit service on different Linux distributions, one by one.

Fluent Bit Installation on Ubuntu

To install td-agent-bit on your Ubuntu server you will have to add the Fluent Bit repository to your Ubuntu OS's sources list so that when we run the install command the td-agent-bit package can be easily downloaded from the 3rd party repository.

Also, to access the 3rd party repository we will need to have the public key to download the required packages from it, so let's start by adding the public key for the Fluent Bit repository server. Run the following command to do so,

$ wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

This will add the GPG Key of the Fluent Bit repository server to your keyring.

Then we will add the URL of the Fluent Bit repository to our sources list available at /etc/apt/sources.list. For this you must have root access.

Run the following command to open the file in vim editor, or any editor of your choice,

vi /etc/apt/sources.list

Press i to start INERT mode in vim editor. and add the following line at the end:

deb https://packages.fluentbit.io/ubuntu/bionic bionic main

If you have Ubuntu Xenial, add the following line:

deb https://packages.fluentbit.io/ubuntu/bionic bionic main

Once you are done, save the sources list file by pressing ESCAPE and then typing !wq followed by Enter. This is the command to write and close any opened file in Vim editor.

Now that we have added a new repository, we will run the apt command to update the apt database.

$ sudo apt-get update

Now we will run the apt-get command to download and install td-agent-bit service on our Ubuntu machine.

$ sudo apt-get install td-agent-bit

To see the commands to start the service, check status of the service and where the configuration files are located, go to the next heading of this tutorial(skip the Installation in CentOS and Debian).

Fluent Bit Installation on Debian

The installation steps for downloading and installing td-agent-bit on Debian OS is same as that of Ubuntu. In Debian Linux distribution too, we use the apt-get command to manage packages.

The only difference is that in the sources list, you will have to add the following line at the end:

For Debian 10 (Buster)

deb https://packages.fluentbit.io/debian/buster buster main

For Debian 9 (Stretch)

deb https://packages.fluentbit.io/debian/stretch stretch main

For Debian 8 (Jessie)

deb https://packages.fluentbit.io/debian/jessie jessie main

Rest every step remains the same.

Fluent Bit Installation on CentOS

In case of CentOS or Redhat Linux distribution, yum command is used to install any new service on these machines. Fluent Bit also has a Yum repository which we will be adding to our machine's yum repository database.

To do so, create a new file with name td-agent-bit.repo in the directory /etc/yum.repos.d/ using any text editor of your choice. We will be using VIM editor.

Go to the specified directory:

cd /etc/yum.repos.d/

Create and open a file with name td-agent-bit.repo

vi td-agent-bit.repo

Press i to start INERT mode in vim editor. and paste the following text to the file:

[td-agent-bit]
name = TD Agent Bit
baseurl = https://packages.fluentbit.io/centos/7/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1

Once you are done, save the file by pressing ESCAPE and then typing !wq followed by Enter.

Then run the following yum command to download and install td-agent-bit service on your machine,

yum install td-agent-bit

Start td-agent-bit Service

Once you have successfully installed the td-agent-bit service on your machine, you can run the following command to start it.

$ sudo service td-agent-bit start

This is same for all the Linux distributions mentioned above.

Then to check the status of the td-agent-bit service, you can run the following command.

$ sudo service td-agent-bit status

You should see the output similar to this for the status command:


td-agent-bit.service - TD Agent Bit
Loaded: loaded (/lib/systemd/system/td-agent-bit.service; disabled; vendor preset: enabled)
Active: active (running) since miƩ 2016-07-06 16:58:25 CST; 2h 45min ago
Main PID: 6739 (td-agent-bit)
Tasks: 1
Memory: 656.0K
CPU: 1.393s
CGroup: /system.slice/td-agent-bit.service
??6739 /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
...

TD Agent Bit Configuration Files

The configuration files for td-agent-bit service can be found in /etc/td-agent-bit/ directory which can be used to set basic configurations like specifying the INPUT plugin, OUTPUT destinations, PARSER, FILTER etc.

You will find the following configuration files there:

  1. parsers.conf - This file has some sample regex PARSERS defined for parsing Apache logs, Apache error logs, Nginx logs etc. We will learn in upcoming tutorials how we can define parser for parsing logs and for other data collection.

  2. plugins.conf

  3. td-agent-bit.conf - This configuration file has all the main configurations like SERVICE, INPUT, OUTPUT, FILTER etc.

By default, Fluent Bit is configured to read the CPU metrics and publish them on standard output. In the next tutorial, we will cover how we can setup Fluent Bit, Elasticsearch and Kibana to aggregate logs, store logs in Elasticsearch and visualize them using Kibana.

If you liked this article, do subscribe to our newsletter, you will find the form on the right sidebar.


RELATED POSTS



Subscribe and receive amazing posts directly in your inbox.