Configure Log Collection, Target, and Visualization - GCP

To get a production ready log configuration you can use tools from the Kubernetes ecosystem and GCP Cloud Logging. In this guide we show you how to set up:

These tools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The GCP Logs Explorer is a monitoring service built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. This guide doesn't describe these tools' functionality in details as it is outside the scope of this guide.

Prerequisite

Before setting up log collection, make sure your Usage Engine Private Edition was installed with JSON formatted logging enabled.

log:
  # Format can be "json" or "raw". Default is "raw"
  format: json

1. Add Fluent helm repository and update repository to retrieve the latest version.

   helm repo add fluent https://fluent.github.io/helm-charts
   helm repo update

2. Retrieve the Elastic Search access credentials by using commands below. Save the output, you will need them in the next step.

   kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
   kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

3. Create a custom values yaml, for example fluent-bit-values.yaml and set the content below. After that, replace values of HTTP_User and HTTP_Passwd to the output from previous step.

   config:
     inputs: |
       [INPUT]
           Name                tail
           Alias               kube_containers
           Tag                 kube_<namespace_name>_<pod_name>_<container_name>
           Exclude_Path        /var/log/containers/*_kube-system_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log,/var/log/containers/*_gmp-system_*.log,/var/log/containers/*_gke-managed-cim_*.log
           Path                /var/log/containers/*.log
           multiline.parser    docker, cri
           Mem_Buf_Limit       50MB
           Skip_Long_Lines     On
           Refresh_Interval    1
           Read_from_Head      True
   
     filters: |
       [FILTER]
           Name                kubernetes
           Match               kube.*
           Kube_URL            https://kubernetes.default.svc:443
           Kube_Tag_Prefix     application.var.log.containers.
           Merge_Log           On
           K8S-Logging.Parser  On
           K8S-Logging.Exclude Off
           Labels              Off
           Annotations         Off
           Use_Kubelet         On
           Kubelet_Port        10250
           Buffer_Size         0
   
     outputs: |
       [OUTPUT]
           Name                es
           Match               *
           Host                elasticsearch-master
           tls                 on
           tls.verify          off
           HTTP_User           elastic
           HTTP_Passwd         zUqEBtrE4H9bfO8K
           Suppress_Type_Name  On
           Index               fluentbit
           Trace_Error         on

4. Install Fluent-bit with the custom values yaml.

   helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml

5. Verify Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.

   kubectl logs <fluent-bit pod name> -n logging

Install Kibana

1. Install Kibana. Note that service type is set to LoadBalancer in order to allow public access.

   helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster. 

Up to this stage, all pods under namespace logging should be up and running. 

NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          4d3h
elasticsearch-master-1           1/1     Running   0          4d3h
elasticsearch-master-2           1/1     Running   0          4d3h
fluent-bit-2kpgr                 1/1     Running   0          3d
fluent-bit-6wtnr                 1/1     Running   0          3d
fluent-bit-ns42z                 1/1     Running   0          3d
kibana-kibana-658dc749cd-hbc8s   1/1     Running   0          3d4h

If all looks good, you can proceed to login to Kibana dashboard web UI.

1. Retrieve the public access IP of the Kibana dashboard.

   kubectl get service -n logging kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

2. Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section.
3. Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous section.
4. 
5. If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.
6. 
7. Go to Management > Stack Management > Kibana. Create Data view matching the index pattern
8. 
9. Go to Analytics > Discover to view logs.
10.  
11. User can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.
12.