Versions Compared

Version Old Version 1 New Version Current
Changes made by

Wei Siong Siow

Wei Siong Siow

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Note

Note!

You need to have a proper Kubernetes cluster setup in order to proceed with these steps.

By default, Usage Engine deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. However, persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates.

...

Info

Example - Installing Elasticsearch without Persistent storage

This example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage:

https://github.com/elastic/helm-charts/tree/master/elasticsearch

Code Block
helm install elasticsearch elastic/elasticsearch -n amazon-cloudwatchlogging --set=persistence.enabled=false

...

Kibana will be installed to the same namespace as Fluent-bit, i.e., logging. 

Install Kibana. Note  

Note that service type is set to LoadBalancer NodePort to allow public access outside of the cluster.

Code Block
helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80NodePort

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster. 

...

Code Block
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          17h
elasticsearch-master-1           1/1     Running   0          17h
elasticsearch-master-2           1/1     Running   0          17h
fluent-bit-b65kn                 1/1     Running   0          23m
fluent-bit-cdpjg                 1/1     Running   0          23m
fluent-bit-czwmz                 1/1     Running   0          23m
fluentkibana-kibana-bit8446b87c9f-kwrtr                 1/1     Running   0          23m
fluent-bit-rlb7k                 1/1     Running   0          23m
kibana-kibana-8446b87c9f-hrsc8   1hrsc8   1/1     Running   0          80s

If all looks good, you can proceed to login to Kibana dashboard web UI.

...

  1. You can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.

...

  1. A log record in json format is parsed into fields, as below:

...

  1. format is parsed into fields, as below:

Code Block
{
  "@timestamp": [
    "2024-0710-24T0609T10:0237:4428.833Z389Z"
  ],
  "kubernetes.container_hash": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f1b1208cf87d69c71190f56759f1a791660807ad4ea97e014e64a16f51865818b"
  ],
  "kubernetes.container_hash.keyword": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f1b1208cf87d69c71190f56759f1a791660807ad4ea97e014e64a16f51865818b"
  ],
  "kubernetes.container_image": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.23.0-featurebugfix-t-stratus-XE-1369314225-azurefix-helmCICD-chartissue-2024072208512020240924113401-71796fea5d7ef52da9e282-operator"
  ],
  "kubernetes.container_image.keyword": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.23.0-featurebugfix-t-stratus-XE-1369314225-azurefix-helmCICD-chartissue-2024072208512020240924113401-71796fea5d7ef52da9e282-operator"
  ],
  "kubernetes.container_name": [
    "manager"
  ],
  "kubernetes.container_name.keyword": [
    "manager"
  ],
  "kubernetes.docker_id": [
    "83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f140c60cf0501822f9639866d0d237803d77e75312137ccd329b0927d56a97e00e"
  ],
  "kubernetes.docker_id.keyword": [
    "83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f140c60cf0501822f9639866d0d237803d77e75312137ccd329b0927d56a97e00e"
  ],
  "kubernetes.host": [
    "akslima-internalrancher-23883603-vmss000000desktop"
  ],
  "kubernetes.host.keyword": [
    "akslima-internalrancher-23883603-vmss000000desktop"
  ],
  "kubernetes.namespace_name": [
    "uepe"
  ],
  "kubernetes.namespace_name.keyword": [
    "uepe"
  ],
  "kubernetes.pod_id": [
    "14492863a52fc196-e495e4be-4ea24492-a1c28f6d-7e937c492bd0e73d517e8a8a"
  ],
  "kubernetes.pod_id.keyword": [
    "14492863a52fc196-e495e4be-4ea24492-a1c28f6d-7e937c492bd0e73d517e8a8a"
  ],
  "kubernetes.pod_name": [
    "uepe-operator-controller-manager-795b5d8dd65d7976f9b6-kgpqs7w4t8"
  ],
  "kubernetes.pod_name.keyword": [
    "uepe-operator-controller-manager-795b5d8dd65d7976f9b6-kgpqs7w4t8"
  ],
  "log": [
    "{\"level\":\"info\",\"ts\":\"2024-0710-24T0609T10:0237:44Z28Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2ec1\",\"accumulated duration\":0.202696261039129083}\n"
  ],
  "log_processed.accumulated duration": [
    0.20269626039129082
  ],
  "log_processed.ECDeployment": [
    "uepe/ecd-http2ec1"
  ],
  "log_processed.ECDeployment.keyword": [
    "uepe/ecd-http2ec1"
  ],
  "log_processed.level": [
    "info"
  ],
  "log_processed.level.keyword": [
    "info"
  ],
  "log_processed.logger": [
    "controllers.ECDeployment"
  ],
  "log_processed.logger.keyword": [
    "controllers.ECDeployment"
  ],
  "log_processed.msg": [
    "Finished reconciling"
  ],
  "log_processed.msg.keyword": [
    "Finished reconciling"
  ],
  "log_processed.ts": [
    "2024-0710-24T0609T10:0237:4428.000Z"
  ],
  "log.keyword": [
    "{\"level\":\"info\",\"ts\":\"2024-0710-24T0609T10:0237:44Z28Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2ec1\",\"accumulated duration\":0.202696261039129083}\n"
  ],
  "stream": [
    "stderr"
  ],
  "stream.keyword": [
    "stderr"
  ],
  "time": [
    "2024-0710-24T0609T10:0237:4428.833Z389Z"
  ],
  "_id": "ixdV45AB-BIbU7-jvbNvOJXacJIBnq_iaVq27Nny",
  "_index": "fluentbit",
  "_score": null
}