...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
| Note |
|---|
Note!You need to have a proper GKE cluster setup in order to proceed with these steps. Refer to Set Up Kubernetes Cluster - GCP (4.13) to create the GKE cluster first. |
By default Usage Engine deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. But persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates.
To get a production ready log configuration you can use tools from the Kubernetes ecosystem and GCP Cloud Logging. In this guide we show you how to set up:
GCP Cloud logging for storage and monitoring
Fluent-bit for log collection and log forwarding
Elasticsearch for log storage
Kibana for log visualization
Thesetools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The GCP The GCP Logs Explorer is a monitoring service built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. This guide doesn't describe these tools' functionality in details as it is outside the scope of this guide.
Prerequisite
Beforesetting up log collection, make sure your Usage Engine Private Edition was installed with JSON with JSON formatted logging enabled.
| Code Block |
|---|
...
log:
# Format can be "json" or "raw". Default is "raw"
format: json |
Stream container logs to GCP Cloud Logging
Before using GCP Cloud Logging, you need to ensure Cloud Logging API is enabled on your Google Cloud project. Refer to
...
the guide https://cloud.google.com/kubernetes-engine/docs/troubleshooting/logging to verify if logging is enabled.
Fluent-bit is a log processor that used to
...
send containers logs to GCP Cloud Logging.
...
By default, a
...
managed Fluent-bit will be installed by GKE during cluster creation.
After Cloud Logging API is enabled, all containers logs should automatically send to the Cloud Logging. To verify logging, go to GCP console page Logging > Logs Explorer and check if container logs are populated.
...
...
Stream container logs
...
to Elastic Search and visualize with Kibana
| Note |
|---|
Note!Note that you must install Elastic Search, Fluent-bit and Kibana on the same namespace in order to allow working properly. There are some of the reasons:
Hence, in this guide we are using namespace 'logging' for the installations. |
Install Elastic Search
Add Elastic Search repository to Helm and update repository to retrieve the latest version.
Code Block
...
helm repo add elastic https://helm.elastic.co helm repo updateInstall Elastic Search.
Note title Note! For simplicity this example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage:
https://github.com/elastic/helm-charts/tree/master/elasticsearchCode Block
...
helm install elasticsearch elastic/elasticsearch -n logging --set=persistence.enabled=false
Install custom Fluent-bit
Add Fluent helm repository and update repository to retrieve the latest version.
Code Block
...
helm repo add fluent https://fluent.github.io/helm-charts helm repo updateRetrieve the Elastic Search access credentials by using commands below. Save the output, you will need them in the next step.
Code Block
...
kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -dCreate a custom values yaml, for
...
example fluent-bit-values.yaml and set the content below. After that, replace values of HTTP_User and HTTP_Passwd to the output from previous step.
Code Block
...
config: inputs: | [INPUT] Name tail Alias kube_containers Tag kube_<namespace_name>_<pod_name>_<container_name> Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log,/var/log/containers/*_gmp-system_*.log,/var/log/containers/*_gke-managed-cim_*.log Path /var/log/containers/*.log multiline.parser docker, cri Mem_Buf_Limit 50MB Skip_Long_Lines On Refresh_Interval 1 Read_from_Head True filters: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_Tag_Prefix application.var.log.containers. Merge_Log On K8S-Logging.Parser On K8S-Logging.Exclude Off Labels Off Annotations Off Use_Kubelet On Kubelet_Port 10250 Buffer_Size 0 outputs: | [OUTPUT] Name es Match * Host elasticsearch-master tls on tls.verify off HTTP_User elastic HTTP_Passwd zUqEBtrE4H9bfO8K Suppress_Type_Name On Index fluentbit Trace_Error onInstall Fluent-bit with the custom values yaml.
Code Block
...
helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yamlVerify Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.
Code Block
...
kubectl logs <fluent-bit pod name> -n logging
Install Kibana
Install Kibana. Note that service type is set to LoadBalancer in order to allow public access.
Code Block
...
helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80
Configure Kibana
Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster.
Up to this stage, all pods under namespace logging should be up and running.
| Code Block |
|---|
NAME READY STATUS RESTARTS AGE elasticsearch-master-0 1/1 Running 0 4d3h elasticsearch-master-1 1/1 Running 0 4d3h elasticsearch-master-2 1/1 Running 0 4d3h fluent-bit-2kpgr 1/1 Running 0 3d fluent-bit-6wtnr 1/1 Running 0 3d fluent-bit-ns42z 1/1 Running 0 3d kibana-kibana-658dc749cd-hbc8s 1/1 Running 0 3d4h |
...
If all looks good, you can proceed to login to Kibana dashboard web UI.
Retrieve the public access IP of the Kibana dashboard.
Code Block
...
kubectl get service -n logging kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].ip}'Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section.
Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous section.
...
If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.
...
Go to Management > Stack Management > Kibana. Create Data view matching the index pattern
...
Go to Analytics> Discover to view logs.
...
User can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.
...
