New Tutorials:   TKINTER    KOTLIN    JAVASCRIPT    SASS/SCSS    PL/SQL    Matplotlib    C++ Programs
CLOSE
   Fluent Bit  Elasticsearch  AWS  
   Technology    Cloud

Configure Fluent Bit with AWS Elasticsearch Service

         
 AUGUST 2, 2020   by iamabhishek

Fluent Bit is a data collector service which can be used for collecting data from IoT sensors, or logs from applications running in cloud cluster like Kubernetes, etc.

Fluent Bit with Kibana and Elasticsearch is popularly used for log collection, aggregation and visualization. You can either setup all these 3 services on a Linux server, in a Kubernetes cluster running services within containers or we can even have Fluent Bit running on a Linux machine or Kubernetes cluster(EKS) while Elasticsearch running on AWS.

Why use AWS Elasticsearch?

AWS Elasticsearch is a fully managed Elasticsearch service provided by AWS, where we can configure the number of data nodes requried, number of master nodes required and what type of instance (can be different for both data and master node) to run these nodes and we can start our Elasticsearch cluster.

If you want to use EFK stack for log collection and analysis on production, you should use AWS Elasticsearch service because it's difficult to scale the Elasticsearch service, if you plan to do it yourself.

Rather, have Elasticsearch AWS service, with which Kibana service comes as free plugin, and run a log collection/aggregation service at your end which can be Fluent Bit, Fluentd or Logstash or any other similar service.

When you have Elasticsearch running on AWS, there are two ways to connect it with Fluent Bit service:

By allowing open access for any service to be able to access Elasticsearch service, if the new service is running within the EKS cluster on AWS.

Or, we need to have an AWS IAM role with External Id to access the Elasticsearch service because Fluent Bit service uses the assumeRole API to get access to Elasticsearch running on AWS.

For this introductory tutorial, we will be make our Elasticsearch service to allow open access to the domain, because we have the Fluent Bit running in EKS cluster as Daemonset.

Fluent Bit es Output Plugin

Fluent Bit service provides us with an es output plugin for elasticsearch service to configure Fluent Bit to send output to the configured Elasticsearch service.

Following are the properties that we will be using while configuring Fluent Bit to push data to AWS Elasticsearch service.

[OUTPUT]
        Name            es
        Match           *
        Host            https://vpc-my-elasticsearch-8erHY26sdqudaybp89HCVZey4ifacliy.ap-south-1.es.amazonaws.com
        Port            443
        AWS_Auth        Off
        AWS_Region      ap-south-1
        tls             On
        Logstash_Format On
        Logstash_Prefix my-logs

The additional AWS related property used are:

Host: Here we will provide the VPC endpoint for our AWS Elasticsearch service which would be like https://vpc-my-elasticsearch-8erHY26sdqudaybp89HCVZey4ifacliy.ap-south-1.es.amazonaws.com which you can also use to access your Elasticsearch service.

Port: Port will be 443. Yes, it will not be 9200. AWS Elasticsearch communicates over SSL port.

AWS_Auth: We have set this property to Off because we are not using any kind of authentication to access our AWS Elasticsearch service.

AWS_Region: The region in which you have started your Elasticsearch service. I had mine running in the ap-south-1 region.

tls: We will set this to On.

Other properties available in Fluent Bit are AWS_Role_ARN which provides the AWS IAM Role to assume to put records to your Amazon ES cluster and AWS_External_ID which is the External ID for the AWS IAM Role specified with aws_role_arn.

We will not be using these two properties.

Enable Open Access for Elasticsearch:

Before enabling open access, make sure you have a security group specifying that only the services within the EKS cluster can access the AWS ES cluster.

Go to AWS ES dashboard:

use fluent bit with AWS elasticsearch

Select your Elasticsearch service and then click on Actions dropdown and choose the option Modify access policy:

use fluent bit with AWS elasticsearch service

And select Allow open access to the domain and you should be good to go.

use fluent bit with AWS elasticsearch service

Once done, your JSON defined access policy should look like the following:


{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:ap-south-1:98191918321:domain/my-elasticsearch/*"
}
]
}

And with this, our Fluent Bit service should start pushing logs or whatever data you are collecting to the AWS Elasticsearch service. You can see the logs by accessing the Kibana UI which comes along the AWS ES service.

If you face any issues, do share in the comment section below.


RELATED POSTS



Subscribe and receive amazing posts directly in your inbox.