By default, Usage Engine deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. However, persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates.

To get a production ready log configuration you can use tools from the Kubernetes ecosystem. In this guide we show you how to set up:

• Fluent-bit for log collection and log forwarding

• Elasticsearch for log storage

• Kibana for log visualization

These tools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data.

Prerequisite

Before setting up log collection, ensure that Usage Engine was installed with JSON formatted logging enabled, as below:

Install Elastic Search

Elastic Search will be installed to the namespace logging

1. Create namespace logging

2. Add Elastic Search repository to Helm and update repository to retrieve the latest version with the following commands:

3. Install Elastic Search. 

Install Fluent-bit

Fluent-bit will be installed in the same namespace as Elastic Search, that is logging.

1. Get the service name of Elastic Search pods with the following command:

This service name is the value set to Host in [OUTPUT] directive.

2. Get the username and password credential for Elastic X-Pack access with the following commands:

The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in the [OUTPUT] directive.

3. Create a custom values yaml file, for example fluent-bit-values.yaml with the following content:

4. Add the fluent helm repo and update repo with the following commands:

5. Deploy the Fluent Bit DaemonSet to the cluster with the following command:

6. Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully with the following command:

Install Kibana

Kibana will be installed to the same namespace as Fluent-bit, i.e., logging. 

Install Kibana. 

Note that service type is set to NodePort to allow access outside of the cluster.

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster. 

Up to this stage, all pods under namespace logging should be up and running. 

If all looks good, you can proceed to login to Kibana dashboard web UI.

1. Retrieve the public access IP Address of the Kibana dashboard with the following command:

2. Login to Kibana dashboard web interface using the HTTP_User and HTTP_Passwd configured in the previous section.

3. Go to Management > Stack Management > Index Management.

4. If the Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.

Open

Index Management

5. Go to Management > Stack Management > Kibana and create a Data view matching the index pattern

Open

Data Views

6. Go to Analytics > Discover to search for logs belong to each index pattern respectively.

7. You can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.

8. A log record in json format is parsed into fields, as below: