...

Note!

You need to have a proper AKS cluster setup in order to proceed with these steps. Refer to Set Up Kubernetes Cluster - Azure (4.3) to create the

...

AKS cluster first.

...

Insert excerpt

	Log Collection
	Log Collection
name	Kubernetes-output
nopanel	true

To get a production ready log configuration you can use tools from the Kubernetes ecosystem and Azure Monitor Logs Log Analytics Service. In this guide we show you how to set up:

Fluent-bit for log collection and log forwarding
Elasticsearch for log storage
Kibana for log visualization
Azure Monitor Logs for Analytics

Stream container logs to Elastic Search and visualize with Kibana

...

Log Analytics for querying monitoring logs

These tools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The Azure Log Analytics is a tool for querying monitoring logs built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. See the official user documentation for detailed information about these tools.

Prerequisite

Insert excerpt

	Log Collection
	Log Collection
name	prereq
nopanel	true

AKS Container Insights

You can collect, store and analyze logs and event data for container debugging purpose. To enable container insights, go to Home > your-cluster-name > Monitoring > Logs. Click Configure monitoring to proceed.

...

Azure Log Analytics

You can log data from Azure Monitor in a Log Analytics workspace. Azure provides an analysis engine and a rich query language. The logs show the context of any problems, and are useful for identifying root causes.

...

For more information, refer to Azure Log Analytics tutorial.

Stream Container Logs to Elastic Search and Visualize with Kibana

Note

Important

You must install Elastic Search, Fluent-bit and Kibana on the same namespace in order to allow working properly. These are some of the reasons:

Elastic Search service needs to be accessible by Fluent-bit and Kibana to establish connection.
Kibana required Elastic Search master cert secret presented on the namespace.

Hence, in this guide we are using namespace 'logging' for the installations.

Install Elastic Search

Elastic Search will be installed to the namespace logging

Create namespace logging

Code Block
kubectl create namespace logging

Add Elastic Search repository to Helm and update repository to retrieve the latest version with the following commands:

Code Block
helm repo add elastic https://helm.elastic.co

Code Block
helm repo update

Install Elastic Search.

Info

Example - Installing Elasticsearch without Persistent storage

This example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage:

https://github.com/elastic/helm-charts/tree/master/elasticsearch

Code Block
helm install elasticsearch elastic/elasticsearch -n amazon-cloudwatch --set=persistence.enabled=false

Install Fluent-bit

Fluent-bit will be installed in the same namespace as Elastic Search, that is logging.

Get the service name of Elastic Search pods with the following command:

Code Block
kubectl get svc -n logging

This service name is the value set to Host in [OUTPUT] directive.

Get the username and password credential for Elastic X-Pack access with the following commands:

Code Block
kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.username}' \| base64 -d

Code Block
kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.password}' \| base64 -d

The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in the [OUTPUT] directive.

Create a custom values yaml file, for example fluent-bit-values.yaml with the following content:

Code Block

config:
  inputs: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        /var/log/containers/kube-proxy*
        Path                /var/log/containers/*.log
        multiline.parser    docker, cri
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      True
  filters: |
    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              Off
        Annotations         Off
        Buffer_Size         0
  outputs: |
    [OUTPUT]
        Name                es
        Match               application.*
        Host                elasticsearch-master
        tls                 On
        tls.verify          Off
        HTTP_User           elastic
        HTTP_Passwd         SbeSsXiuWbAnbxUT
        Suppress_Type_Name  On
        Index               fluentbit
        Trace_Error         On

Add thefluent helm repo and update repo with the following commands:

Code Block
helm repo add fluent https://fluent.github.io/helm-charts

Code Block
helm repo update

Deploy the Fluent Bit DaemonSet to the cluster with the following command:

Code Block
helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml

Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully with the following command:

Code Block
kubectl logs <fluent-bit pod name> -n logging

Install Kibana

Kibana will be installed to the same namespace as Fluent-bit, i.e., logging.

Install Kibana. Note that service type is set to LoadBalancer to allow public access

Code Block
helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster.

Up to this stage, all pods under namespace logging should be up and running.

Code Block

NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          17h
elasticsearch-master-1           1/1     Running   0          17h
elasticsearch-master-2           1/1     Running   0          17h
fluent-bit-b65kn                 1/1     Running   0          23m
fluent-bit-cdpjg                 1/1     Running   0          23m
fluent-bit-czwmz                 1/1     Running   0          23m
fluent-bit-kwrtr                 1/1     Running   0          23m
fluent-bit-rlb7k                 1/1     Running   0          23m
kibana-kibana-8446b87c9f-hrsc8   1/1     Running   0          80s

If all looks good, you can proceed to login to Kibana dashboard web UI.

Retrieve the public access IP Address of the Kibana dashboard with the following command:

Code Block
kubectl get service -n logging kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Login to Kibana dashboard web interface using the HTTP_User and HTTP_Passwd configured in the previous section.
Go to Management > Stack Management > Index Management.
If the Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.

...

Go to Management > Stack Management > Kibana and create a Data view matching the index pattern

...

Go to Analytics > Discover to search for logs belong to each index pattern respectively.

...

You can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.

...

A log record in json format is parsed into fields, as below:

Code Block

{
  "_p": [
    "F"
  ],
  "_p.keyword": [
    "F"
  ],
  "@timestamp": [
    "2024-07-24T06:02:44.833Z"
  ],
  "kubernetes.container_hash": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f"
  ],
  "kubernetes.container_hash.keyword": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f"
  ],
  "kubernetes.container_image": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.2.0-feature-t-stratus-XE-13693-azure-helm-chart-20240722085120-71796fea5d7-operator"
  ],
  "kubernetes.container_image.keyword": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.2.0-feature-t-stratus-XE-13693-azure-helm-chart-20240722085120-71796fea5d7-operator"
  ],
  "kubernetes.container_name": [
    "manager"
  ],
  "kubernetes.container_name.keyword": [
    "manager"
  ],
  "kubernetes.docker_id": [
    "83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f1"
  ],
  "kubernetes.docker_id.keyword": [
    "83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f1"
  ],
  "kubernetes.host": [
    "aks-internal-23883603-vmss000000"
  ],
  "kubernetes.host.keyword": [
    "aks-internal-23883603-vmss000000"
  ],
  "kubernetes.namespace_name": [
    "uepe"
  ],
  "kubernetes.namespace_name.keyword": [
    "uepe"
  ],
  "kubernetes.pod_id": [
    "14492863-e495-4ea2-a1c2-7e937c492bd0"
  ],
  "kubernetes.pod_id.keyword": [
    "14492863-e495-4ea2-a1c2-7e937c492bd0"
  ],
  "kubernetes.pod_name": [
    "uepe-operator-controller-manager-795b5d8dd6-kgpqs"
  ],
  "kubernetes.pod_name.keyword": [
    "uepe-operator-controller-manager-795b5d8dd6-kgpqs"
  ],
  "log": [
    "{\"level\":\"info\",\"ts\":\"2024-07-24T06:02:44Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2\",\"accumulated duration\":0.202696261}"
  ],
  "log_processed.accumulated duration": [
    0.20269626
  ],
  "log_processed.ECDeployment": [
    "uepe/ecd-http2"
  ],
  "log_processed.ECDeployment.keyword": [
    "uepe/ecd-http2"
  ],
  "log_processed.level": [
    "info"
  ],
  "log_processed.level.keyword": [
    "info"
  ],
  "log_processed.logger": [
    "controllers.ECDeployment"
  ],
  "log_processed.logger.keyword": [
    "controllers.ECDeployment"
  ],
  "log_processed.msg": [
    "Finished reconciling"
  ],
  "log_processed.msg.keyword": [
    "Finished reconciling"
  ],
  "log_processed.ts": [
    "2024-07-24T06:02:44.000Z"
  ],
  "log.keyword": [
    "{\"level\":\"info\",\"ts\":\"2024-07-24T06:02:44Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2\",\"accumulated duration\":0.202696261}"
  ],
  "stream": [
    "stderr"
  ],
  "stream.keyword": [
    "stderr"
  ],
  "time": [
    "2024-07-24T06:02:44.833Z"
  ],
  "_id": "ixdV45AB-BIbU7-jvbNv",
  "_index": "fluentbit",
  "_score": null
}

Version	Old Version 5	New Version Current
Changes made by	Kam Heng Choy	Jenni Wallin
Saved on	Jul 23, 2024	Oct 11, 2024

Page Comparison

Versions Compared

Key

Stream container logs to Elastic Search and visualize with Kibana

Prerequisite

AKS Container Insights

Azure Log Analytics

Stream Container Logs to Elastic Search and Visualize with Kibana

Install Elastic Search

Install Fluent-bit

Install Kibana

Configure Kibana