Configure Log Collection, Target, and Visualization - AWS(4.0)

To get a production ready log configuration you can use tools from the Kubernetes ecosystem and AWS CloudWatch. In this guide we show you how to set up:

These tools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The AWS CloudWatch is a monitoring service built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. This guide doesn't describe these tools' functionality in details as it is outside the scope of this guide.

Enable JSON Logging

Before setting up log collection, make sure that JSON formatted logging is enabled in Usage Engine helm chart.

Download the helm chart and unpack it in local directory.

helm fetch usage-engine-private-edition/usage-engine-private-edition --untar

log:
  # Format can be "json" or "raw". Default is "raw"
  format: json

These are additional steps to configure fluent-bit ConfigMap named fluent-bit-config.

1. Get service name of Elastic Search pods. This service name is the value set to Host in [OUTPUT] directive. 

   kubectl get svc -n amazon-cloudwatch

2. Get username and password credential for Elastic X-Pack access. The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.

   kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
   kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

3. Download fluent-bit deamonset yaml file in local directory

   curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml > fluent-bit.yaml

4. Edit fluent-bit.yaml. Go to ConfigMap named fluent-bit-config. For each config file, add output directive to send logs to Elastic Search
   application-log.conf

       [OUTPUT]
           Name                es
           Match               application.*
           Host                elasticsearch-master
           tls                 on
           tls.verify          off
           HTTP_User           elastic
           HTTP_Passwd         DbrfdbnzCNYympQZ
           Suppress_Type_Name  On
           Index               fluentbit.app

5. dataplane-log.conf

       [OUTPUT]
           Name                es
           Match               dataplane.*
           Host                elasticsearch-master
           tls                 on
           tls.verify          off
           HTTP_User           elastic
           HTTP_Passwd         DbrfdbnzCNYympQZ
           Suppress_Type_Name  On
           Index               fluentbit.dataplane

6. host-log.conf

       [OUTPUT]
           Name                es
           Match               host.*
           Host                elasticsearch-master
           tls                 on
           tls.verify          off
           HTTP_User           elastic
           HTTP_Passwd         DbrfdbnzCNYympQZ
           Suppress_Type_Name  On
           Index               fluentbit.host

7. Delete existing fluent-bit pods, config map.

   kubectl delete -f fluent-bit.yaml

8. Install and apply new configuration to fluent-bit pods, config map  

   kubectl apply -f fluent-bit.yaml

9. Re-associate the IAM role to cloudwatch-agent and fluent-bit service accounts. Replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts.

   kubectl annotate serviceaccounts fluent-bit -n amazon-cloudwatch "eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME"

10. Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.

    kubectl logs <fluent-bit pod name> -n amazon-cloudwatch

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster. 

Up to this stage, all pods under namespace amazon-cloudwatch should be up and running. 

NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          4d3h
elasticsearch-master-1           1/1     Running   0          4d3h
fluent-bit-2kpgr                 1/1     Running   0          3d
fluent-bit-6wtnr                 1/1     Running   0          3d
fluent-bit-ns42z                 1/1     Running   0          3d
kibana-kibana-658dc749cd-hbc8s   1/1     Running   0          3d4h

If all looks good, you can proceed to login to Kibana dashboard web UI.

1. Retrieve the public access hostname of the Kibana dashboard.

   kubectl get service -n amazon-cloudwatch kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

2. Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section
3. Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous section
4. If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically 
5. Go to Management > Stack Management > Kibana. Create Data view matching the index pattern
6. Go to Analytics > Discover to search for logs belong to each index pattern respectively. 
7. User can filter logs using KQL syntax. For instance, enter "kubernetes.pod_name:platform-0" in the KQL filter input field
8. Log record in json format is parsed into fields

   {
     "_p": [
       "F"
     ],
     "_p.keyword": [
       "F"
     ],
     "@timestamp": [
       "2024-02-21T09:14:49.079Z"
     ],
     "kubernetes.container_hash": [
       "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848"
     ],
     "kubernetes.container_hash.keyword": [
       "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848"
     ],
     "kubernetes.container_image": [
       "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator"
     ],
     "kubernetes.container_image.keyword": [
       "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator"
     ],
     "kubernetes.container_name": [
       "manager"
     ],
     "kubernetes.container_name.keyword": [
       "manager"
     ],
     "kubernetes.docker_id": [
       "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d"
     ],
     "kubernetes.docker_id.keyword": [
       "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d"
     ],
     "kubernetes.host": [
       "ip-192-168-34-51.ap-southeast-2.compute.internal"
     ],
     "kubernetes.host.keyword": [
       "ip-192-168-34-51.ap-southeast-2.compute.internal"
     ],
     "kubernetes.namespace_name": [
       "uepe"
     ],
     "kubernetes.namespace_name.keyword": [
       "uepe"
     ],
     "kubernetes.pod_id": [
       "5a911c45-d2b0-4f53-b474-ae8aee304d4a"
     ],
     "kubernetes.pod_id.keyword": [
       "5a911c45-d2b0-4f53-b474-ae8aee304d4a"
     ],
     "kubernetes.pod_name": [
       "uepe-operator-controller-manager-6fdc476cb5-9282q"
     ],
     "kubernetes.pod_name.keyword": [
       "uepe-operator-controller-manager-6fdc476cb5-9282q"
     ],
     "log": [
       "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}"
     ],
     "log_processed.ECDeployment": [
       "uepe/http2"
     ],
     "log_processed.ECDeployment.keyword": [
       "uepe/http2"
     ],
     "log_processed.level": [
       "info"
     ],
     "log_processed.level.keyword": [
       "info"
     ],
     "log_processed.logger": [
       "controllers.ECDeployment"
     ],
     "log_processed.logger.keyword": [
       "controllers.ECDeployment"
     ],
     "log_processed.msg": [
       "Reconciling"
     ],
     "log_processed.msg.keyword": [
       "Reconciling"
     ],
     "log_processed.ts": [
       "2024-02-21T09:14:49.000Z"
     ],
     "log.keyword": [
       "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}"
     ],
     "stream": [
       "stderr"
     ],
     "stream.keyword": [
       "stderr"
     ],
     "time": [
       "2024-02-21T09:14:49.079Z"
     ],
     "_id": "ijvyyo0B9xu2H_IDTAqi",
     "_index": "fluentbit.app",
     "_score": null
   }