...
Note!
You need to have a proper AKS cluster setup in order to proceed with these steps. Refer to Set Up Kubernetes Cluster - Azure (4.3) to create the
...
AKS cluster first.
...
By default Usage Engine deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. But persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates.
To get a production ready log configuration you can use tools from the Kubernetes ecosystem and Azure Log Analytics Service. In this guide we show you how to set up:
Fluent-bit for log collection and log forwarding
Elasticsearch for log storage
Kibana for log visualization
Azure Log Analytics for Analyze
Azure Log Analytics
...
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|
To get a production ready log configuration you can use tools from the Kubernetes ecosystem and Azure Log Analytics Service. In this guide we show you how to set up:
Fluent-bit for log collection and log forwarding
Elasticsearch for log storage
Kibana for log visualization
Azure Log Analytics for querying monitoring logs
These tools give you powerful and flexible log collection, storage, monitoring and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. The Azure Log Analytics is a tool for querying monitoring logs built for DevOps engineers, developers, site reliability engineers (SREs), IT managers, and product owners. See the official user documentation for detailed information about these tools.
Prerequisite
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|
AKS Container Insights
You can collect, store and analyze logs and event data for container debugging purpose. To enable container insights, go to Home > your-cluster-name > Monitoring > Logs. Click Configure monitoring to proceed.
...
Azure Log Analytics
You can log data from Azure Monitor in a Log Analytics workspace. Azure provides an analysis engine and a rich query language. The logs show the context of any problems, and are useful for identifying root causes.
...
For more information, please refer to Azure Log Analytics tutorial.
...
Stream Container Logs to Elastic Search and
...
Visualize with Kibana
| Note |
|---|
Important Note that you You must install Elastic Search, Fluent-bit and Kibana on the same namespace in order to allow working properly. These are some of the reasons:
Hence, in this guide we are using namespace 'logging' for the installations. |
Install Elastic Search
Elastic Search will be installed to the namespace logging
...
Add Elastic Search repository to Helm and update repository to retrieve the latest version with the following commands:
| Code Block |
|---|
helm repo add elastic https://helm.elastic.co |
| Code Block |
|---|
helm repo update |
Install Elastic Search.
| Note | |||
|---|---|---|---|
| Info | |||
Example - Installing Elasticsearch without Persistent storage This example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage: https://github.com/elastic/helm-charts/tree/master/elasticsearch
|
Install Fluent-bit
Fluent-bit will be installed to in the same namespace as Elastic Search, i.e., logging.that is logging.
Get the service name of Elastic Search pods . with the following command:
| Code Block |
|---|
kubectl get svc -n logging |
This service name is the value set to Host in [
...
| Code Block |
|---|
kubectl get svc -n logging |
...
OUTPUT] directive.
Get the username and password credential for Elastic X-Pack access . The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.with the following commands:
| Code Block |
|---|
kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d |
| Code Block |
|---|
kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -dd |
The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in the [OUTPUT] directive.
Create a custom values yaml file, for example fluent-bit-values.yaml with the following content:
| Code Block |
|---|
config:
inputs: |
[INPUT]
Name tail
Tag application.*
Exclude_Path /var/log/containers/kube-proxy*
Path /var/log/containers/*.log
multiline.parser docker, cri
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head True
filters: |
[FILTER]
Name kubernetes
Match application.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix application.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Labels Off
Annotations Off
Buffer_Size 0
outputs: |
[OUTPUT]
Name es
Match application.*
Host elasticsearch-master
tls On
tls.verify Off
HTTP_User elastic
HTTP_Passwd SbeSsXiuWbAnbxUT
Suppress_Type_Name On
Index fluentbit
Trace_Error On |
To add Add the
fluenthelm repo and update repo , runwith the following commands:
| Code Block |
|---|
helm repo add fluent https://fluent.github.io/helm-charts |
...
Deploy the Fluent Bit DaemonSet to the cluster with the following command:
| Code Block |
|---|
helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml |
Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully .with the following command:
| Code Block |
|---|
kubectl logs <fluent-bit pod name> -n logging |
Install Kibana
Kibana will be installed to the same namespace as Fluent-bit, i.e., logging.
...
| Code Block |
|---|
helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80 |
Configure Kibana
Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster.
...
Retrieve the public access IP Address of the Kibana dashboard with the following command:
| Code Block |
|---|
kubectl get service -n logging kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].ip}' |
Login to Kibana dashboard web UI with username password same as HTTPinterface using the HTTP_User and HTTPand HTTP_Passwd configured in the previous section.
Go to Management > Stack Management > Index Management.
If the Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.
...
Go to Management > Stack Management > Kibana. Create and create a Data view matching the index pattern
...
Go to Analytics > Discover to search for logs belong to each index pattern respectively.
...
User You can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.
...
Log A log record in json format is parsed into fields, as below:
| Code Block |
|---|
{
"_p": [
"F"
],
"_p.keyword": [
"F"
],
"@timestamp": [
"2024-07-24T06:02:44.833Z"
],
"kubernetes.container_hash": [
"464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f"
],
"kubernetes.container_hash.keyword": [
"464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f"
],
"kubernetes.container_image": [
"464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.2.0-feature-t-stratus-XE-13693-azure-helm-chart-20240722085120-71796fea5d7-operator"
],
"kubernetes.container_image.keyword": [
"464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.2.0-feature-t-stratus-XE-13693-azure-helm-chart-20240722085120-71796fea5d7-operator"
],
"kubernetes.container_name": [
"manager"
],
"kubernetes.container_name.keyword": [
"manager"
],
"kubernetes.docker_id": [
"83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f1"
],
"kubernetes.docker_id.keyword": [
"83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f1"
],
"kubernetes.host": [
"aks-internal-23883603-vmss000000"
],
"kubernetes.host.keyword": [
"aks-internal-23883603-vmss000000"
],
"kubernetes.namespace_name": [
"uepe"
],
"kubernetes.namespace_name.keyword": [
"uepe"
],
"kubernetes.pod_id": [
"14492863-e495-4ea2-a1c2-7e937c492bd0"
],
"kubernetes.pod_id.keyword": [
"14492863-e495-4ea2-a1c2-7e937c492bd0"
],
"kubernetes.pod_name": [
"uepe-operator-controller-manager-795b5d8dd6-kgpqs"
],
"kubernetes.pod_name.keyword": [
"uepe-operator-controller-manager-795b5d8dd6-kgpqs"
],
"log": [
"{\"level\":\"info\",\"ts\":\"2024-07-24T06:02:44Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2\",\"accumulated duration\":0.202696261}"
],
"log_processed.accumulated duration": [
0.20269626
],
"log_processed.ECDeployment": [
"uepe/ecd-http2"
],
"log_processed.ECDeployment.keyword": [
"uepe/ecd-http2"
],
"log_processed.level": [
"info"
],
"log_processed.level.keyword": [
"info"
],
"log_processed.logger": [
"controllers.ECDeployment"
],
"log_processed.logger.keyword": [
"controllers.ECDeployment"
],
"log_processed.msg": [
"Finished reconciling"
],
"log_processed.msg.keyword": [
"Finished reconciling"
],
"log_processed.ts": [
"2024-07-24T06:02:44.000Z"
],
"log.keyword": [
"{\"level\":\"info\",\"ts\":\"2024-07-24T06:02:44Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2\",\"accumulated duration\":0.202696261}"
],
"stream": [
"stderr"
],
"stream.keyword": [
"stderr"
],
"time": [
"2024-07-24T06:02:44.833Z"
],
"_id": "ixdV45AB-BIbU7-jvbNv",
"_index": "fluentbit",
"_score": null
} |