Configure Log Collection, Target, and Visualization - GCP

Note!

You need to have a proper GKE cluster setup in order to proceed with these steps. Refer to Set Up Kubernetes Cluster - GCP to create the GKE cluster first.

By default Usage Engine deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. But persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates.

To get a production ready log configuration you can use tools from the Kubernetes ecosystem and GCP Cloud Logging. In this guide we show you how to set up:

Fluent-bit for log collection and log forwarding
Elasticsearch for log storage
Kibana for log visualization
GCP Logs Explorer for monitoring

These tools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The GCP Logs Explorer is a monitoring service built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. This guide doesn't describe these tools' functionality in details as it is outside the scope of this guide.

Prerequisite

Before setting up log collection, make sure your Usage Engine Private Edition was installed with JSON formatted logging enabled.

log:
  # Format can be "json" or "raw". Default is "raw"
  format: json

You also need to make sure Cloud Logging API is enabled in your Google Cloud project. In case Cloud Logging API is disabled in your project, follow the guide https://cloud.google.com/kubernetes-engine/docs/troubleshooting/logging to enable it.

Install Fluent-bit

Fluent-bit is used to stream containers logs to GCP Cloud Logging. In this guide, we are using the default Fluent Bit that was pre-installed and managed by GKE cluster. If you need to change the default behaviour, please refer to https://cloud.google.com/kubernetes-engine/docs/concepts/about-logs#custom_agents for guidance.

Install Elastic Search

Note that you should install both Elastic Search and Kibana on the same namespace, mainly due to Kibana installation required Elastic Search master cert secret to be presented. Therefore, in this guide namespace 'logging' was used.

Add Elastic Search repository to Helm and update repository to retrieve the latest version.
```
helm repo add elastic https://helm.elastic.co
helm repo update
```
Install Elastic Search.
Note!

For simplicity this example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage:

https://github.com/elastic/helm-charts/tree/master/elasticsearch
```
helm install elasticsearch elastic/elasticsearch -n logging --set=persistence.enabled=false
```

Install Kibana

Download the Kibana helm chart and unpack it in local directory.
```
helm fetch elastic/kibana --untar
```

Change directory to kibana. Edit values.yaml. Add service annotation to create Internet-facing, Network type Load Balancer .

service:
  type: ClusterIP
  loadBalancerIP: ""
  port: 5601
  nodePort: ""
  labels: {}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

Install Kibana by path to an unpacked local directory

helm install kibana kibana -n logging --set=service.type=LoadBalancer --set=service.port=80

Configure Fluent-bit to send logs to Elastic Search

These are additional steps to configure fluent-bit ConfigMap named fluent-bit-config.

Get service name of Elastic Search pods. This service name is the value set to Host in [OUTPUT] directive.
```
kubectl get svc -n amazon-cloudwatch
```

Get username and password credential for Elastic X-Pack access. The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.

kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

Download fluent-bit deamonset yaml file in local directory

curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml > fluent-bit.yaml

Edit fluent-bit.yaml. Go to ConfigMap named fluent-bit-config. For each config file, add output directive to send logs to Elastic Search

application-log.conf

    [OUTPUT]
        Name                es
        Match               application.*
        Host                elasticsearch-master
        tls                 on
        tls.verify          off
        HTTP_User           elastic
        HTTP_Passwd         DbrfdbnzCNYympQZ
        Suppress_Type_Name  On
        Index               fluentbit.app

dataplane-log.conf

    [OUTPUT]
        Name                es
        Match               dataplane.*
        Host                elasticsearch-master
        tls                 on
        tls.verify          off
        HTTP_User           elastic
        HTTP_Passwd         DbrfdbnzCNYympQZ
        Suppress_Type_Name  On
        Index               fluentbit.dataplane

host-log.conf

    [OUTPUT]
        Name                es
        Match               host.*
        Host                elasticsearch-master
        tls                 on
        tls.verify          off
        HTTP_User           elastic
        HTTP_Passwd         DbrfdbnzCNYympQZ
        Suppress_Type_Name  On
        Index               fluentbit.host

Delete existing fluent-bit pods, config map.
```
kubectl delete -f fluent-bit.yaml
```
Install and apply new configuration to fluent-bit pods, config map
```
kubectl apply -f fluent-bit.yaml
```
Re-associate the IAM role to cloudwatch-agent and fluent-bit service accounts. Replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts.
```
kubectl annotate serviceaccounts fluent-bit -n amazon-cloudwatch "eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME"
```
Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.
```
kubectl logs <fluent-bit pod name> -n amazon-cloudwatch
```

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster.

Up to this stage, all pods under namespace amazon-cloudwatch should be up and running.

NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          4d3h
elasticsearch-master-1           1/1     Running   0          4d3h
fluent-bit-2kpgr                 1/1     Running   0          3d
fluent-bit-6wtnr                 1/1     Running   0          3d
fluent-bit-ns42z                 1/1     Running   0          3d
kibana-kibana-658dc749cd-hbc8s   1/1     Running   0          3d4h

If all looks good, you can proceed to login to Kibana dashboard web UI.

Retrieve the public access hostname of the Kibana dashboard.

kubectl get service -n amazon-cloudwatch kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section
Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous section
If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically
Go to Management > Stack Management > Kibana. Create Data view matching the index pattern
Go to Analytics > Discover to search for logs belong to each index pattern respectively.
User can filter logs using KQL syntax. For instance, enter "kubernetes.pod_name:platform-0" in the KQL filter input field

Log record in json format is parsed into fields

{
  "_p": [
    "F"
  ],
  "_p.keyword": [
    "F"
  ],
  "@timestamp": [
    "2024-02-21T09:14:49.079Z"
  ],
  "kubernetes.container_hash": [
    "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848"
  ],
  "kubernetes.container_hash.keyword": [
    "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848"
  ],
  "kubernetes.container_image": [
    "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator"
  ],
  "kubernetes.container_image.keyword": [
    "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator"
  ],
  "kubernetes.container_name": [
    "manager"
  ],
  "kubernetes.container_name.keyword": [
    "manager"
  ],
  "kubernetes.docker_id": [
    "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d"
  ],
  "kubernetes.docker_id.keyword": [
    "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d"
  ],
  "kubernetes.host": [
    "ip-192-168-34-51.ap-southeast-2.compute.internal"
  ],
  "kubernetes.host.keyword": [
    "ip-192-168-34-51.ap-southeast-2.compute.internal"
  ],
  "kubernetes.namespace_name": [
    "uepe"
  ],
  "kubernetes.namespace_name.keyword": [
    "uepe"
  ],
  "kubernetes.pod_id": [
    "5a911c45-d2b0-4f53-b474-ae8aee304d4a"
  ],
  "kubernetes.pod_id.keyword": [
    "5a911c45-d2b0-4f53-b474-ae8aee304d4a"
  ],
  "kubernetes.pod_name": [
    "uepe-operator-controller-manager-6fdc476cb5-9282q"
  ],
  "kubernetes.pod_name.keyword": [
    "uepe-operator-controller-manager-6fdc476cb5-9282q"
  ],
  "log": [
    "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}"
  ],
  "log_processed.ECDeployment": [
    "uepe/http2"
  ],
  "log_processed.ECDeployment.keyword": [
    "uepe/http2"
  ],
  "log_processed.level": [
    "info"
  ],
  "log_processed.level.keyword": [
    "info"
  ],
  "log_processed.logger": [
    "controllers.ECDeployment"
  ],
  "log_processed.logger.keyword": [
    "controllers.ECDeployment"
  ],
  "log_processed.msg": [
    "Reconciling"
  ],
  "log_processed.msg.keyword": [
    "Reconciling"
  ],
  "log_processed.ts": [
    "2024-02-21T09:14:49.000Z"
  ],
  "log.keyword": [
    "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}"
  ],
  "stream": [
    "stderr"
  ],
  "stream.keyword": [
    "stderr"
  ],
  "time": [
    "2024-02-21T09:14:49.079Z"
  ],
  "_id": "ijvyyo0B9xu2H_IDTAqi",
  "_index": "fluentbit.app",
  "_score": null
}