Page Comparison

To get a production ready log configuration you can use tools from the Kubernetes ecosystem and GCP Cloud Logging. In this guide we show you how to set up:

Thesetools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The GCP Logs Explorer is a monitoring service built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. This guide doesn't describe these tools' functionality in details as it is outside the scope of this guide.

Prerequisite

Before setting up log collection, make sure your Usage Engine Private Edition was installed with JSON formatted logging enabled.

log:
  # Format can be "json" or "raw". Default is "raw"
  format: json

...

1. Add Fluent helm repository and update repository to retrieve the latest version.

   Code Block
   linenumbers true
   helm repo add fluent https://fluent.github.io/helm-charts helm repo update

   
   

2. Retrieve the Elastic Search access credentials by using commands below. Save the output, you will need them in the next step.

   Code Block
   linenumbers true
   kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

   
   

3. Create a custom values yaml, for example fluent-bit-values.yaml and set the content below. After that, replace values of HTTP_User and HTTP_Passwd to the output from previous step.

   Code Block
   linenumbers true
   config: inputs: | [INPUT] Name tail Alias kube_containers Tag kube_<namespace_name>_<pod_name>_<container_name> Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log,/var/log/containers/*_gmp-system_*.log,/var/log/containers/*_gke-managed-cim_*.log Path /var/log/containers/*.log multiline.parser docker, cri Mem_Buf_Limit 50MB Skip_Long_Lines On Refresh_Interval 1 Read_from_Head True filters: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_Tag_Prefix application.var.log.containers. Merge_Log On K8S-Logging.Parser On K8S-Logging.Exclude Off Labels Off Annotations Off Use_Kubelet On Kubelet_Port 10250 Buffer_Size 0 outputs: | [OUTPUT] Name es Match * Host elasticsearch-master tls on tls.verify off HTTP_User elastic HTTP_Passwd zUqEBtrE4H9bfO8K Suppress_Type_Name On Index fluentbit Trace_Error on

   
   

4. Install Fluent-bit with the custom values yaml.

   Code Block
   linenumbers true
   helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml

   
   

5. Verify Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.

   Code Block
   linenumbers true
   kubectl logs <fluent-bit pod name> -n logging

   
   

Install Kibana

1. Install Kibana. Note that service type is set to LoadBalancer in order to allow public access.

   Code Block
   linenumbers true
   helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80

   
   

...

1. Retrieve the public access IP of the Kibana dashboard.

   Code Block
   linenumbers true
   kubectl get service -n logging kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

   
   

2. Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section.
3. Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous section.
4. 
5. If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.
6. 
7. Go to Management > Stack Management > Kibana. Create Data view matching the index pattern
8. 
9. Go to Analytics > Discover to view logs.
10.  Image Modified
11. User can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.
12. Image Modified