To get a production ready log configuration you can use tools from the Kubernetes ecosystem and AWS CloudWatch. In this guide we show you how to set up:

• Fluent-bit for log collection and log forwarding

• Elasticsearch for log storage

• Kibana for log visualization

• AWS CloudWatch for monitoring

These tools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The AWS CloudWatch is a monitoring service built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. See the official user documentation for detailed information about these tools.

Prerequisite

Setup AWS IAM OIDC Provider

To use AWS Identity and Access Management (IAM) roles for service accounts, an IAM OIDC provider must exist for your cluster's OIDC issuer URL. Prior to creating AWS policy and role, you need to setup Identity Provider using EKS cluster's OpenID Connect Provider URL.

1. Login to AWS Management Console, go to EKS > Clusters > Your Cluster Name.

2. On Overview tab, section Details, click on the copy button under OpenID Connect Provider URL to copy the URL to the clipboard.

   Open

   

   AWS Console - Overview tab

3. Go to IAM > Identity Providers.

4. Add an Identity Provider and select OpenID Connect.

5. Paste the copied URL as Provider URL.

6. Enter "sts.amazonaws.com" as Audience.

7. Click Add Provider and proceed to complete the Identity Providers creation 

Setup AWS IAM Policy and Role

In order for Fluent-bit to send logs to AWS CloudWatch, you need to setup AWS access policy to access the AWS CloudWatch and attach this policy to an AWS Role. 

1. Login to AWS Management Console, go to IAM > Policies.

2. Create a new policy using the JSON tab in the Policy Editor. Enter the permission statement in JSON format, as below:

   { "Version": "2012-10-17", "Statement": [ { "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "*", "Effect": "Allow" } ] }

3. Click Next and proceed to create the policy.

4. Back in the IAM Dashboard, go to IAM > Roles.

5. Create a new role and select Web Identity.

6. Select the OpenID Connect Provider Id as Identity Provider.

7. Click Next and proceed to create the role.

Once the new role has been created, you need to to edit the role's trust relationship to associate it to the Fluent-bit's Service Account.

1. Go to IAM > Roles > Your Role Name.

2. On the Trust relationship tab, edit trust policy.

3. Edit the "StringEquals" field to use the Fluent-bit's namespace and Service Account Name, as below:

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::211006581866:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/360F8C7227656FC5627D5DA70F181583" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.ap-southeast-2.amazonaws.com/id/360F8C7227656FC5627D5DA70F181583:sub": "system:serviceaccount:<Fluent-bit namespace>:<fluent-bit Service Account Name>" } } } ] }

Install Fluent-bit

To stream container logs to CloudWatch Logs, install AWS for Fluent-bit:

1. Create a namespace called amazon-cloudwatch with the following command:

   kubectl create namespace amazon-cloudwatch

2. Create a ConfigMap called fluent-bit-cluster-info and replace my-cluster-name and my-cluster-region with your cluster's name and Region, as below:

3. Deploy the Fluent-bit DaemonSet to the cluster with the following command:

4. Associate the IAM role to cloudwatch-agent and fluent-bit service accounts and replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts with the following command:

5. Go to CloudWatch > View logs and verify that the following log groups have been created:

6. For each log group, verify there are log streams available in the Log stream tab.

   Open

   

   Log streams

Install Elastic Search

Elastic search will be installed to the same namespace as Fluent-bit, i.e., amazon-cloudwatch. 

1. Add Elastic Search repository to Helm and update repository to retrieve the latest version with the following commands:

2. Install Elastic Search. 

Install Kibana

Kibana will be installed in the same namespace as Fluent-bit, that is amazon-cloudwatch. 

1. Download the Kibana helm chart and unpack it in local directory with the following command:

2. Change directory to kibana and edit the values.yaml file by adding service annotation to create an Internet-facing, Network type Load Balancer, as below:

3. Install Kibana by path to an unpacked local directory with the following command:

Configure Fluent-bit to send logs to Elastic Search 

These are additional steps to configure fluent-bit ConfigMap named fluent-bit-config.

1. Get the service name of the Elastic Search pods with the following command:

   This service name is the value set to Host in [OUTPUT] directive. 

2. Get username and password credential for Elastic X-Pack access. with the following commands:

   The decrypted username and password are the values set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.

3. Download the fluent-bit deamonset yaml file into a local directory with the following command:

4. Edit the fluent-bit.yaml file by going to the ConfigMap named fluent-bit-config, and for each config file, add the output directive to send logs to Elastic Search, as below:

   application-log.conf

   dataplane-log.conf

5. host-log.conf

6. Delete the existing fluent-bit pods, config map with the following command:

7. Install and apply the new configuration to fluent-bit pods, config map  with the following command: 

8. Re-associate the IAM role with the cloudwatch-agent and fluent-bit service accounts, and replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts with the following command:

9. Verify every Fluent-bit pod's log with the following command:

   You should not see any error or exception if connection to Elastic Search is established successfully.

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster. 

Up to this stage, all pods under namespace amazon-cloudwatch should be up and running. 

If all looks good, you can proceed to login to Kibana dashboard web UI.

1. Retrieve the public access hostname of the Kibana dashboard.

2. Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section

3. Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous section

4. If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.

5. Go to Management > Stack Management > Kibana. Create Data view matching the index pattern.

6. Go to Analytics > Discover to search for logs belong to each index pattern respectively. 

7. User can filter logs using KQL syntax. For instance, enter "kubernetes.pod_name:platform-0" in the KQL filter input field.

8. Log record in json format is parsed into fields.