Page Comparison

To get a production ready log configuration you can use tools from the Kubernetes ecosystem and AWS CloudWatch. In this guide we show you how to set up:

Thesetools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The AWS CloudWatch is a monitoring service built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. This guide doesn't describe these tools' functionality in details as it is outside the scope of this guide.

Prerequisite

Before setting up log collection, make sure your Usage Engine Private Edition was installed with JSON formatted logging enabled.

log:
  # Format can be "json" or "raw". Default is "raw"
  format: json

These are additional steps to configure fluent-bit ConfigMap named fluent-bit-config.

1. Get service name of Elastic Search pods. This service name is the value set to Host in [OUTPUT] directive. 

   
   

   Code Block
   linenumbers true
   kubectl get svc -n amazon-cloudwatch

   
   

2. Get username and password credential for Elastic X-Pack access. The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.

   
   

   Code Block
   linenumbers true
   kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

   
   

3. Download fluent-bit deamonset yaml file in local directory

   
   

   Code Block
   linenumbers true
   curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml > fluent-bit.yaml

   
   

4. Edit fluent-bit.yaml. Go to ConfigMap named fluent-bit-config. For each config file, add output directive to send logs to Elastic Search

   
   

   Code Block
   title application-log.conf linenumbers true
   [OUTPUT] Name es Match application.* Host elasticsearch-master tls on tls.verify off HTTP_User elastic HTTP_Passwd DbrfdbnzCNYympQZ Suppress_Type_Name On Index fluentbit.app

   
   

5. 
   

   Code Block
   title dataplane-log.conf linenumbers true
   [OUTPUT] Name es Match dataplane.* Host elasticsearch-master tls on tls.verify off HTTP_User elastic HTTP_Passwd DbrfdbnzCNYympQZ Suppress_Type_Name On Index fluentbit.dataplane

   
   

6. 
   

   Code Block
   title host-log.conf linenumbers true
   [OUTPUT] Name es Match host.* Host elasticsearch-master tls on tls.verify off HTTP_User elastic HTTP_Passwd DbrfdbnzCNYympQZ Suppress_Type_Name On Index fluentbit.host

   
   

7. Delete existing fluent-bit pods, config map.

   
   

   Code Block
   linenumbers true
   kubectl delete -f fluent-bit.yaml

   
   

8. Install and apply new configuration to fluent-bit pods, config map  

   
   

   Code Block
   linenumbers true
   kubectl apply -f fluent-bit.yaml

   
   

9. Re-associate the IAM role to cloudwatch-agent and fluent-bit service accounts. Replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts.

   
   

   Code Block
   linenumbers true
   kubectl annotate serviceaccounts fluent-bit -n amazon-cloudwatch "eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME"

   
   

10. Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.

    
    

    Code Block
    linenumbers true
    kubectl logs <fluent-bit pod name> -n amazon-cloudwatch

    
    

...

1. Retrieve the public access hostname of the Kibana dashboard.

   
   

   Code Block
   linenumbers true
   kubectl get service -n amazon-cloudwatch kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

   
   

2. Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section
3. Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous sectionImage Modified
4. If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically  Image Modified
5. Go to Management > Stack Management > Kibana. Create Data view matching the index patternImage Modified
6. Go to Analytics > Discover to search for logs belong to each index pattern respectively. Image Modified
7. User can filter logs using KQL syntax. For instance, enter "kubernetes.pod_name:platform-0" in the KQL filter input fieldImage Modified
8. Log record in json format is parsed into fields

   
   

   Code Block

   { "_p": [ "F" ], "_p.keyword": [ "F" ], "@timestamp": [ "2024-02-21T09:14:49.079Z" ], "kubernetes.container_hash": [ "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848" ], "kubernetes.container_hash.keyword": [ "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848" ], "kubernetes.container_image": [ "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator" ], "kubernetes.container_image.keyword": [ "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator" ], "kubernetes.container_name": [ "manager" ], "kubernetes.container_name.keyword": [ "manager" ], "kubernetes.docker_id": [ "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d" ], "kubernetes.docker_id.keyword": [ "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d" ], "kubernetes.host": [ "ip-192-168-34-51.ap-southeast-2.compute.internal" ], "kubernetes.host.keyword": [ "ip-192-168-34-51.ap-southeast-2.compute.internal" ], "kubernetes.namespace_name": [ "uepe" ], "kubernetes.namespace_name.keyword": [ "uepe" ], "kubernetes.pod_id": [ "5a911c45-d2b0-4f53-b474-ae8aee304d4a" ], "kubernetes.pod_id.keyword": [ "5a911c45-d2b0-4f53-b474-ae8aee304d4a" ], "kubernetes.pod_name": [ "uepe-operator-controller-manager-6fdc476cb5-9282q" ], "kubernetes.pod_name.keyword": [ "uepe-operator-controller-manager-6fdc476cb5-9282q" ], "log": [ "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}" ], "log_processed.ECDeployment": [ "uepe/http2" ], "log_processed.ECDeployment.keyword": [ "uepe/http2" ], "log_processed.level": [ "info" ], "log_processed.level.keyword": [ "info" ], "log_processed.logger": [ "controllers.ECDeployment" ], "log_processed.logger.keyword": [ "controllers.ECDeployment" ], "log_processed.msg": [ "Reconciling" ], "log_processed.msg.keyword": [ "Reconciling" ], "log_processed.ts": [ "2024-02-21T09:14:49.000Z" ], "log.keyword": [ "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}" ], "stream": [ "stderr" ], "stream.keyword": [ "stderr" ], "time": [ "2024-02-21T09:14:49.079Z" ], "_id": "ijvyyo0B9xu2H_IDTAqi", "_index": "fluentbit.app", "_score": null }

   
   

...