Configure Log Collection, Target, and Visualization - AWS(3.2)

Note!

Some of the configuration requires the kubectl command. If this is your first time performing an AWS installation, in order to use kubectl, you will have to set up the EKS cluster. Refer to Setup (3.2) to create the EKS cluster first.

By default deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. But persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates.

To get a production ready log configuration you can use tools from the Kubernetes ecosystem. In this guide we show you how to set up:

Fluent for log collection
Elasticsearch for log storage
Kibana for log visualization

These tools give you powerful and flexible log collection, storage, and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. It is outside the scope of this guide to describe those capabilities.

Note!

This guide includes the steps for installing the tools in the Kubernetes cluster. If you deploy in a cloud environment, these tools may already be configured. In that case, it is enough to perform the configuration and skip the installation steps.

Enable JSON Logging in

Before setting up log collection, make sure that JSON formatted logging is enabled in using the setting:

log:
  # Format can be "json" or "raw". Default is "raw"
  format: json

in values.yaml.

Create a Namespace

To isolate the logging functionality from the rest of the system and make it possible to reuse it from multiple instances, a separate namespace is created to host the installed tools.

kubectl create namespace logging

Add Helm Repos

helm repo add elastic https://helm.elastic.co

helm repo add fluent https://fluent.github.io/helm-charts

Install Elasticsearch

helm fetch elastic/elasticsearch --untar

It is recommended to enable Elasticsearch security before proceeding with installation:

https://www.elastic.co/guide/en/elasticsearch/reference/7.11/get-started-enable-security.html?blade=kibanasecuritymessage

Note!

For simplicity this example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage:

https://github.com/elastic/helm-charts/tree/master/elasticsearch

helm upgrade --install elasticsearch elasticsearch -n logging --set=persistence.enabled=false

Install Kibana

helm fetch elastic/kibana --untar

helm upgrade --install kibana kibana -n logging --set=service.type=LoadBalancer --set=service.port=80

Install Fluentd

helm fetch fluent/fluentd --untar

Edit values.yaml to enable System Log collection over syslog. Add the following source to section 01_sources.conf (leaving existing sources unaltered):

01_sources.conf: |-
	<source>
      @type syslog
      port 5140
      bind 0.0.0.0
      tag system
    </source>

Edit values.yaml to specify the JSON based log format:

02_filters.conf: |-
    <label @KUBERNETES>
      <match kubernetes.var.log.containers.fluentd**>
        @type relabel
        @label @FLUENT_LOG
      </match>

      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        skip_labels false
        skip_container_metadata false
        skip_namespace_metadata true
        skip_master_url true
      </filter>

      <filter kubernetes.var.log.containers.**>
        @type parser
        <parse>
          @type json
          json_parser json
        </parse>
        replace_invalid_sequence true
        emit_invalid_record_to_error false
        key_name log
        reserve_data true
        remove_key_name_field true
      </filter>

      <match **>
        @type relabel
        @label @DISPATCH
      </match>
    </label>

helm upgrade --install fluentd fluentd -n logging

Enable System Log forwarding over Syslog

Configure according to:

Log Forwarding(3.0)

The parameters are set in Helm values.yaml file as described in Installation Instructions(3.2).

Verify that all services are up and running

$ kubectl get pod -n logging
NAME                            READY   STATUS    RESTARTS   AGE
elasticsearch-master-0          1/1     Running   0          43m
elasticsearch-master-1          1/1     Running   0          44m
elasticsearch-master-2          1/1     Running   0          44m
fluentd-8g95p                   1/1     Running   0          19h
fluentd-sqv7j                   1/1     Running   0          19h
fluentd-zgx6t                   1/1     Running   0          19h
kibana-kibana-56c9f469d-l7dtv   1/1     Running   0          105m

If all looks good, you can get the URL for the Kibana dashboard with the below command:

kubectl get service -n logging kibana-kibana -o jsonpath={.status.loadBalancer.ingress[0].hostname}

Open the Kibana dashboard and create a new Index Pattern matching the "fluentd" index.

Go to Discover view to search collected log data. For instance, to search the platform log file, enter the search query "kubernetes.pod_name:platform" in the KQL field.

Log records that are properly JSON formatted will be parsed into fields, like:

	"thread": "main",
    "level": "WARN",
    "loggerName": "com.digitalroute.picostart.PlatformClassLoader",
    "marker": {
      "name": "PS",
      "parents": [
        {
          "name": "LIFECYCLE"
        }
      ]
    },
    "message": "Starting Web Server",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
    "instant": {
      "epochSecond": 1614690212,
      "nanoOfSecond": 819721000
    },
    "threadId": 1,
    "threadPriority": 5,
    "timestamp": "2021-03-02T13:03:32.819+0000"

While records that are not JSON formatted will be displayed per row in the "log" field, like:

 "log": "2021-03-02 13:03:35.788:INFO:oejs.Server:pool-10-thread-1: Started @33338ms\n",