Versions Compared

Version Old Version 6 New Version 7
Changes made by

Kam Heng Choy

Kam Heng Choy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
helm install elasticsearch elastic/elasticsearch -n logging --set=persistence.enabled=false

Install Fluent-bit

Fluent-bit will be installed to the same namespace as Elastic Search, i.e., logging.

  1. Get service name of Elastic Search pods. This service name is the value set to Host in [OUTPUT] directive.

Code Block
kubectl get svc -n logging

  1. Get username and password credential for Elastic X-Pack access. The decrypted username and password are the value set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.

Code Block
kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
Code Block
kubectl get secrets --namespace=logging elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

  1. Create a custom values yaml, for example fluent-bit-values.yaml with the following content

Code Block
config:
  inputs: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        /var/log/containers/kube-proxy*
        Path                /var/log/containers/*.log
        multiline.parser    docker, cri
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      True
     
  filters: |
    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              Off
        Annotations         Off
        Buffer_Size         0
  outputs: |
    [OUTPUT]
        Name                es
        Match               application.*
        Host                elasticsearch-master
        tls                 On
        tls.verify          Off
        HTTP_User           elastic
        HTTP_Passwd         SbeSsXiuWbAnbxUT
        Suppress_Type_Name  On
        Index               fluentbit
        Trace_Error         On

  1. To add thefluent helm repo and update repo, run:

Code Block
helm repo add fluent https://fluent.github.io/helm-charts
Code Block
helm repo update

  1. Deploy the Fluent Bit DaemonSet to the cluster

Code Block
helm install fluent-bit fluent/fluent-bit -n logging -f fluent-bit-values.yaml

  1. Verify every Fluent-bit pod's log. Should not see any error or exception if connection to Elastic Search is established successfully.

Code Block
kubectl logs <fluent-bit pod name> -n logging