Versions Compared

Version Old Version 2 New Version 3
Changes made by

Wei Siong Siow

Wei Siong Siow

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info

Example - Installing Elasticsearch without Persistent storage

This example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage:

https://github.com/elastic/helm-charts/tree/master/elasticsearch

Code Block
helm install elasticsearch elastic/elasticsearch -n amazon-cloudwatchlogging --set=persistence.enabled=false

...

Kibana will be installed to the same namespace as Fluent-bit, i.e., logging. 

Install Kibana. Note  

Note that service type is set to LoadBalancer NodePort to allow public access outside of the cluster.

Code Block
helm install kibana elastic/kibana -n logging --set=service.type=LoadBalancer --set=service.port=80NodePort

Configure Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elastic Search cluster. 

...

Code Block
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          17h
elasticsearch-master-1           1/1     Running   0          17h
elasticsearch-master-2           1/1     Running   0          17h
fluent-bit-b65kn                 1/1     Running   0          23m
fluent-bit-cdpjg                 1/1     Running   0          23m
fluent-bit-czwmz                 1/1     Running   0          23m
fluentkibana-kibana-bit-kwrtr              8446b87c9f-hrsc8   1/1     Running   0          23m
fluent-bit-rlb7k                 1/1     Running   0          23m
kibana-kibana-8446b87c9f-hrsc8   1/1     Running   0          80s

...

80s

If all looks good, you can proceed to login to Kibana dashboard web UI.

...

  1. You can filter logs using KQL syntax. For instance, enter "ECDeployment" in the KQL filter input field.

...

  1. A log record in json format is parsed into fields, as below:

...

  1. format is parsed into fields, as below:

Code Block
{
  "@timestamp": [
    "2024-0710-24T0609T10:0237:4428.833Z389Z"
  ],
  "kubernetes.container_hash": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f1b1208cf87d69c71190f56759f1a791660807ad4ea97e014e64a16f51865818b"
  ],
  "kubernetes.container_hash.keyword": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci@sha256:0c76cd048b6540854f0723f3145845511f10ba5bbea8aee198ee21562efb3a8f1b1208cf87d69c71190f56759f1a791660807ad4ea97e014e64a16f51865818b"
  ],
  "kubernetes.container_image": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.23.0-featurebugfix-t-stratus-XE-1369314225-azurefix-helmCICD-chartissue-2024072208512020240924113401-71796fea5d7ef52da9e282-operator"
  ],
  "kubernetes.container_image.keyword": [
    "464113009138.dkr.ecr.eu-west-1.amazonaws.com/mz-ci:4.23.0-featurebugfix-t-stratus-XE-1369314225-azurefix-helmCICD-chartissue-2024072208512020240924113401-71796fea5d7ef52da9e282-operator"
  ],
  "kubernetes.container_name": [
    "manager"
  ],
  "kubernetes.container_name.keyword": [
    "manager"
  ],
  "kubernetes.docker_id": [
    "83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f140c60cf0501822f9639866d0d237803d77e75312137ccd329b0927d56a97e00e"
  ],
  "kubernetes.docker_id.keyword": [
    "83a59c96dfcfa135e728e963b847234f0554859964009b45fe44389fe1e1c0f140c60cf0501822f9639866d0d237803d77e75312137ccd329b0927d56a97e00e"
  ],
  "kubernetes.host": [
    "akslima-internalrancher-23883603-vmss000000desktop"
  ],
  "kubernetes.host.keyword": [
    "akslima-internalrancher-23883603-vmss000000desktop"
  ],
  "kubernetes.namespace_name": [
    "uepe"
  ],
  "kubernetes.namespace_name.keyword": [
    "uepe"
  ],
  "kubernetes.pod_id": [
    "14492863a52fc196-e495e4be-4ea24492-a1c28f6d-7e937c492bd0e73d517e8a8a"
  ],
  "kubernetes.pod_id.keyword": [
    "14492863a52fc196-e495e4be-4ea24492-a1c28f6d-7e937c492bd0e73d517e8a8a"
  ],
  "kubernetes.pod_name": [
    "uepe-operator-controller-manager-795b5d8dd65d7976f9b6-kgpqs7w4t8"
  ],
  "kubernetes.pod_name.keyword": [
    "uepe-operator-controller-manager-795b5d8dd65d7976f9b6-kgpqs7w4t8"
  ],
  "log": [
    "{\"level\":\"info\",\"ts\":\"2024-0710-24T0609T10:0237:44Z28Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2ec1\",\"accumulated duration\":0.202696261039129083}\n"
  ],
  "log_processed.accumulated duration": [
    0.20269626039129082
  ],
  "log_processed.ECDeployment": [
    "uepe/ecd-http2ec1"
  ],
  "log_processed.ECDeployment.keyword": [
    "uepe/ecd-http2ec1"
  ],
  "log_processed.level": [
    "info"
  ],
  "log_processed.level.keyword": [
    "info"
  ],
  "log_processed.logger": [
    "controllers.ECDeployment"
  ],
  "log_processed.logger.keyword": [
    "controllers.ECDeployment"
  ],
  "log_processed.msg": [
    "Finished reconciling"
  ],
  "log_processed.msg.keyword": [
    "Finished reconciling"
  ],
  "log_processed.ts": [
    "2024-0710-24T0609T10:0237:4428.000Z"
  ],
  "log.keyword": [
    "{\"level\":\"info\",\"ts\":\"2024-0710-24T0609T10:0237:44Z28Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Finished reconciling\",\"ECDeployment\":\"uepe/ecd-http2ec1\",\"accumulated duration\":0.202696261039129083}\n"
  ],
  "stream": [
    "stderr"
  ],
  "stream.keyword": [
    "stderr"
  ],
  "time": [
    "2024-0710-24T0609T10:0237:4428.833Z389Z"
  ],
  "_id": "ixdV45AB-BIbU7-jvbNvOJXacJIBnq_iaVq27Nny",
  "_index": "fluentbit",
  "_score": null
}