Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Note!

You need to have a proper EKS cluster setup in order to proceed with these steps. Refer to Set Up Kubernetes Cluster - AWS (4.3) to create the EKS cluster first.

...

Setup AWS IAM OIDC Provider

To use AWS Identity and Access Management (IAM) roles for service accounts, an IAM OIDC provider must exist for your cluster's OIDC issuer URL. Prior to creating AWS policy and role, you need to setup Identity Provider using EKS cluster's OpenID Connect Provider URL.

  1. Login to AWS Management Console, go to EKS > Clusters > Your Cluster Name.

  2. On Overview tab, section Details, click on the copy button under OpenID Connect Provider URL to copy the URL to the clipboard.

    overview.png
  3. Go to IAM > Identity Providers.

  4. Add an Identity Provider and select OpenID Connect.

  5. Paste the copied URL as Provider URL.

  6. Enter "sts.amazonaws.com" as Audience.

  7. Click Add Provider and proceed to complete the Identity Providers creation 

Setup AWS IAM Policy and Role

...

  1. Go to IAM > Roles > Your Role Name.

  2. On the Trust relationship tab, edit trust policy.

  3. Edit the "StringEquals" field to use the Fluent-bit's namespace and Service Account Name, as below:

    Code Block
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": "arn:aws:iam::211006581866:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/360F8C7227656FC5627D5DA70F181583"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "oidc.eks.ap-southeast-2.amazonaws.com/id/360F8C7227656FC5627D5DA70F181583:sub": "system:serviceaccount:<Fluent-bit namespace>:<fluent-bit Service Account Name>"
                    }
                }
            }
        ]
    }

Install Fluent-bit

To stream container logs to CloudWatch Logs, install AWS for Fluent-bit:

  1. Create a namespace called amazon-cloudwatch with the following command:

    Code Block
    kubectl create namespace amazon-cloudwatch
  2. Create a ConfigMap called fluent-bit-cluster-info and replace my-cluster-name and my-cluster-region with your cluster's name and Region, as below:

    Code Block
    ClusterName=<my-cluster-name>
    RegionName=<my-cluster-region>
    FluentBitHttpPort='2020'
    FluentBitReadFromHead='Off'
    [[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
    [[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
    kubectl create configmap fluent-bit-cluster-info \
    --from-literal=cluster.name=${ClusterName} \
    --from-literal=http.server=${FluentBitHttpServer} \
    --from-literal=http.port=${FluentBitHttpPort} \
    --from-literal=read.head=${FluentBitReadFromHead} \
    --from-literal=read.tail=${FluentBitReadFromTail} \
    --from-literal=logs.region=${RegionName} -n amazon-cloudwatch
  3. Deploy the Fluent-bit DaemonSet to the cluster with the following command:

    Code Block
    kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
  4. Associate the IAM role to cloudwatch-agent and fluent-bit service accounts and replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts with the following command:

    Code Block
    kubectl annotate serviceaccounts fluent-bit -n amazon-cloudwatch "eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME"
  5. Go to CloudWatch > View logs and verify that the following log groups have been created:

    Code Block
    /aws/containerinsights/Your Cluster Name/application
    /aws/containerinsights/Your Cluster Name/dataplane
    /aws/containerinsights/Your Cluster Name/host
  6. For each log group, verify there are log streams available in the Log stream tab.

    log-streams.png

...

  1. Get the service name of the Elastic Search pods with the following command:

    Code Block
    kubectl get svc -n amazon-cloudwatch

    This service name is the value set to Host in [OUTPUT] directive. 

  2. Get username and password credential for Elastic X-Pack access. with the following commands:

    Code Block
    kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
    Code Block
    kubectl get secrets --namespace=amazon-cloudwatch elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

    The decrypted username and password are the values set to HTTP_User and HTTP_Passwd in [OUTPUT] directive.

  3. Download the fluent-bit deamonset yaml file into a local directory with the following command:

    Code Block
    curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml > fluent-bit.yaml
  4. Edit the fluent-bit.yaml file by going to the ConfigMap named fluent-bit-config, and for each config file, add the output directive to send logs to Elastic Search, as below:

    application-log.conf

    Code Block
        [OUTPUT]
            Name                es
            Match               application.*
            Host                elasticsearch-master
            tls                 on
            tls.verify          off
            HTTP_User           elastic
            HTTP_Passwd         DbrfdbnzCNYympQZ
            Suppress_Type_Name  On
            Index               fluentbit.app

    dataplane-log.conf

  5. Code Block
        [OUTPUT]
            Name                es
            Match               dataplane.*
            Host                elasticsearch-master
            tls                 on
            tls.verify          off
            HTTP_User           elastic
            HTTP_Passwd         DbrfdbnzCNYympQZ
            Suppress_Type_Name  On
            Index               fluentbit.dataplane

    host-log.conf

    Code Block
        [OUTPUT]
            Name                es
            Match               host.*
            Host                elasticsearch-master
            tls                 on
            tls.verify          off
            HTTP_User           elastic
            HTTP_Passwd         DbrfdbnzCNYympQZ
            Suppress_Type_Name  On
            Index               fluentbit.host
  6. Delete the existing fluent-bit pods, config map with the following command:

    Code Block
    kubectl delete -f fluent-bit.yaml
  7. Install and apply the new configuration to fluent-bit pods, config map  with the following command: 

    Code Block
    kubectl apply -f fluent-bit.yaml
  8. Re-associate the IAM role with the cloudwatch-agent and fluent-bit service accounts, and replace ACCOUNT_ID and IAM_ROLE_NAME with AWS Account ID and the IAM role used for service accounts with the following command:

    Code Block
    kubectl annotate serviceaccounts fluent-bit -n amazon-cloudwatch "eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME"
  9. Verify every Fluent-bit pod's log with the following command:

    Code Block
    kubectl logs <fluent-bit pod name> -n amazon-cloudwatch

    You should not see any error or exception if connection to Elastic Search is established successfully.

...

  1. Retrieve the public access hostname of the Kibana dashboard.

    Code Block
    kubectl get service -n amazon-cloudwatch kibana-kibana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
  2. Login to Kibana dashboard web UI with username password same as HTTP_User and HTTP_Passwd configured in previous section

  3. Go to Management > Stack Management > Index Management. Create the Index Template with Index Pattern matching the indexes configured in previous section

    index-templates.png
  4. If Fluent-bit connection to Elastic Search established successfully, the Indices is created automatically.

    indices.png
  5. Go to Management > Stack Management > Kibana. Create Data view matching the index pattern.

    data-views.png
  6. Go to Analytics > Discover to search for logs belong to each index pattern respectively. 

    discover.png
  7. User can filter logs using KQL syntax. For instance, enter "kubernetes.pod_name:platform-0" in the KQL filter input fieldImage Removed.

    discover-search.pngImage Added
  8. Log record in json format is parsed into fields.

    Code Block
    {
      "_p": [
        "F"
      ],
      "_p.keyword": [
        "F"
      ],
      "@timestamp": [
        "2024-02-21T09:14:49.079Z"
      ],
      "kubernetes.container_hash": [
        "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848"
      ],
      "kubernetes.container_hash.keyword": [
        "ghcr.io/digitalroute-public/usage-engine-private-edition@sha256:fceb32e07cfae86db58d9a83328e4539eb5f42455cd6a0463e9ac955b3642848"
      ],
      "kubernetes.container_image": [
        "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator"
      ],
      "kubernetes.container_image.keyword": [
        "ghcr.io/digitalroute-public/usage-engine-private-edition:4.0.0-operator"
      ],
      "kubernetes.container_name": [
        "manager"
      ],
      "kubernetes.container_name.keyword": [
        "manager"
      ],
      "kubernetes.docker_id": [
        "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d"
      ],
      "kubernetes.docker_id.keyword": [
        "9af8ba62db2aacbb39435ed8894bc078013ea1126a561a85a1d486ee8e12367d"
      ],
      "kubernetes.host": [
        "ip-192-168-34-51.ap-southeast-2.compute.internal"
      ],
      "kubernetes.host.keyword": [
        "ip-192-168-34-51.ap-southeast-2.compute.internal"
      ],
      "kubernetes.namespace_name": [
        "uepe"
      ],
      "kubernetes.namespace_name.keyword": [
        "uepe"
      ],
      "kubernetes.pod_id": [
        "5a911c45-d2b0-4f53-b474-ae8aee304d4a"
      ],
      "kubernetes.pod_id.keyword": [
        "5a911c45-d2b0-4f53-b474-ae8aee304d4a"
      ],
      "kubernetes.pod_name": [
        "uepe-operator-controller-manager-6fdc476cb5-9282q"
      ],
      "kubernetes.pod_name.keyword": [
        "uepe-operator-controller-manager-6fdc476cb5-9282q"
      ],
      "log": [
        "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}"
      ],
      "log_processed.ECDeployment": [
        "uepe/http2"
      ],
      "log_processed.ECDeployment.keyword": [
        "uepe/http2"
      ],
      "log_processed.level": [
        "info"
      ],
      "log_processed.level.keyword": [
        "info"
      ],
      "log_processed.logger": [
        "controllers.ECDeployment"
      ],
      "log_processed.logger.keyword": [
        "controllers.ECDeployment"
      ],
      "log_processed.msg": [
        "Reconciling"
      ],
      "log_processed.msg.keyword": [
        "Reconciling"
      ],
      "log_processed.ts": [
        "2024-02-21T09:14:49.000Z"
      ],
      "log.keyword": [
        "{\"level\":\"info\",\"ts\":\"2024-02-21T09:14:49Z\",\"logger\":\"controllers.ECDeployment\",\"msg\":\"Reconciling\",\"ECDeployment\":\"uepe/http2\"}"
      ],
      "stream": [
        "stderr"
      ],
      "stream.keyword": [
        "stderr"
      ],
      "time": [
        "2024-02-21T09:14:49.079Z"
      ],
      "_id": "ijvyyo0B9xu2H_IDTAqi",
      "_index": "fluentbit.app",
      "_score": null
    }