Configure Log Collection, Target, and Visualization - OCI

By default Usage Engine deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. But persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates.

• Fluent-bit for log collection and log forwarding
• Elasticsearch for log storage
• Kibana for log visualization
• OCI Logging Analytics Service for Analytics
  

log:
  # Format can be "json" or "raw". Default is "raw"
  format: json

Kubernetes Monitoring Solution in Oracle Logging Analytics

Use the Kubernetes Monitoring Solution inOracle Logging Analyticsto monitor and generate insights into your Kubernetes deployed in OCI, third party public clouds, private clouds, or on-premises including managed Kubernetes deployments.

To connect your Kubernetes cluster with Logging Analytics:

1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Solutions, and click Kubernetes. The Kubernetes Monitoring Solution page opens.

2. In the Kubernetes Monitoring Solution page, click Connect clusters. The Add Data wizard opens. Here, the Monitor Kubernetes section is already expanded. Click Oracle OKE. The Configure OKE environment monitoring page opens.

3. Select the OKE cluster that you want to connect withOracle Logging Analytics and click Next
4. Select the compartment for telemetry data and related monitoring resources.
5. Do not select the required Policies and dynamic groups. 
6. Select the metrics server for the collection of usage metrics. You can disable the check box if you have already installed it.
7. Select the Solution deployment option to enable manual deployment of the selected cluster
8. Click Configure log collection to proceed
9. Wait for the log collection configuration to complete
10. Complete and proceed to the Log Explorer.

Install Elastic Search

Elastic search will be installed to the namespace logging. 

1. Create namespace logging

   kubectl create namespace logging

2. Add Elastic Search repository to Helm and update repository to retrieve the latest version

   helm repo add elastic https://helm.elastic.co
   helm repo update

3. Install Elastic Search. 

   Note!

   For simplicity this example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage:
   
   https://github.com/elastic/helm-charts/tree/master/elasticsearch

   helm install elasticsearch elastic/elasticsearch -n logging --set=persistence.enabled=false

Install Fluent-bit

Fluent-bit will be installed to the same namespace as Elastic Search, i.e., logging.

1. Get service name of Elastic Search pods. This service name is the value set to Host in [OUTPUT] directive.

   kubectl get svc -n logging

2. Get username and password credential for Elastic X-Pack access. The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.

   kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
   kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

3. Create a custom values yaml, for example fluent-bit-values.yaml with the following content

   config:
     inputs: |
       [INPUT]
           Name                tail
           Tag                 application.*
           Exclude_Path        /var/log/containers/kube-proxy*
           Path                /var/log/containers/*.log
           multiline.parser    docker, cri
           Mem_Buf_Limit       50MB
           Skip_Long_Lines     On
           Refresh_Interval    10
           Read_from_Head      True
       
     filters: |
       [FILTER]
           Name                kubernetes
           Match               application.*
           Kube_URL            https://kubernetes.default.svc:443
           Kube_Tag_Prefix     application.var.log.containers.
           Merge_Log           On
           Merge_Log_Key       log_processed
           K8S-Logging.Parser  On
           K8S-Logging.Exclude Off
           Labels              Off
           Annotations         Off
           Buffer_Size         0
     outputs: |
       [OUTPUT]
           Name                es
           Match               application.*
           Host                elasticsearch-master
           tls                 On
           tls.verify          Off
           HTTP_User           elastic
           HTTP_Passwd         SbeSsXiuWbAnbxUT
           Suppress_Type_Name  On
           Index               fluentbit
           Trace_Error         On
   
   

4. To add thefluent helm repo, run:

   helm repo add fluent https://fluent.github.io/helm-charts
   helm repo update

5. Deploy the Fluent Bit DaemonSet to the cluster.

   helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml

6. Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.

   kubectl logs <fluent-bit pod name> -n logging

Install Kibana

Kibana will be installed to the same namespace as Fluent-bit, i.e., logging. 

1. Install Kibana. Note that service type is set to LoadBalancer to allow public access.

   helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80