Configure Log Collection, Target, and Visualization - AWS
Note!
Some of the configuration requires the kubectl command. If this is your first time performing an AWS Usage Engine installation, in order to use kubectl, you will have to set up the EKS cluster. Refer to Setup - AWS Terraform (2.3) to create the EKS cluster first.
By default Usage Engine deployed in Kubernetes outputs logging to disk and console output. If persistent disk storage is enabled, the logs end up on the mounted shared disk. But persistent disk is not always the desired log target, especially in a cloud environment where persistent data is typically accessed through services and APIs rather than as files. The console logs can be accessed through the "kubectl logs" command or from a Kubernetes dashboard. The buffer for storing the Kubernetes console logs is stored in memory only though and thus will be lost when a Pod terminates. To get a production ready log configuration you can use tools from the Kubernetes ecosystem. In this guide we show you how to set up: These tools give you powerful and flexible log collection, storage, and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. It is outside the scope of this guide to describe those capabilities. Note! This guide includes the steps for installing the tools in the Kubernetes cluster. If you deploy Usage Engine in a cloud environment, these tools may already be configured. In that case, it is enough to perform the configuration and skip the installation steps. Before setting up log collection, make sure that JSON formatted logging is enabled in Usage Engine using the setting: in values.yaml. To isolate the logging functionality from the rest of the system and make it possible to reuse it from multiple Usage Engine instances, a separate namespace is created to host the installed tools. It is recommended to enable Elasticsearch security before proceeding with installation: Note! For simplicity this example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage: This is done to open port 5140 for receiving systemlog data and to accept JSON format logs from Kubernetes pods. Also, you the certificates must be mounted for elasticsearch so that Fluentd Edit the fluentd values.yaml file (/home/mzadmin/pe/fluentd/values.yaml) to enable System Log collection over syslog. Replace the services part of the file to define a clusterIP service running on port 5140 protocol UDP. This will handle the MZ systemlog and later modify the platform to send logs to port 5140. In the same file, edit the volumes and volumeMounts part to mount the elasticsearch certificates (created during the helm install elasticsearch). Also, add a new volume to mount this in the In the same file change the settings for the liveness and readiness probes to increase the failure threshold. The VM is not that quick and the default settings will mean that fluentd exits and restarts because it's not responding quick enough. Edit the same values.yaml file and add the following <source> to In the same file, edit the Edit the same file and modify the hostname in the Install fluentd with the command below: Configure according to: Log Forwarding Fluentd connects to elasticsearch and is ready ( If all looks good, you can get the URL for the Kibana dashboard with the below command: Open the Kibana dashboard and create a new Index Pattern matching the "fluentd" index. Go to Discover view to search collected log data. For instance, to search the platform log file, enter the search query "kubernetes.pod_name:platform" in the KQL field. Log records that are properly JSON formatted will be parsed into fields, like: While records that are not JSON formatted will be displayed per row in the "log" field, like:Enable JSON Logging in Usage Engine
log:
# Format can be "json" or "raw". Default is "raw"
format: json
Create a Namespace
kubectl create namespace logging
Add Helm Repos
helm repo add elastic https://helm.elastic.co
helm repo add fluent https://fluent.github.io/helm-charts
Install Elasticsearch
helm fetch elastic/elasticsearch --untar
https://github.com/elastic/helm-charts/tree/master/elasticsearchhelm upgrade --install elasticsearch elasticsearch -n logging --set=persistence.enabled=false
Install Kibana
helm fetch elastic/kibana --untar
helm upgrade --install kibana kibana -n logging --set=service.type=LoadBalancer --set=service.port=80
Install Fluentd
helm fetch fluent/fluentd --untar
Updating the Default settings
## Fluentd service
##
service:
type: "ClusterIP"
annotations: {}
ports:
- name: "syslog"
protocol: UDP
containerPort: 5140
/elasticsearch
directory. volumes:
…
- name: elasticsearch-master-certs
secret:
secretName: elasticsearch-master-certs
volumeMounts:
…
- name: elasticsearch-master-certs
mountPath: /elasticsearch
# Configure the livenessProbe
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
livenessProbe:
httpGet:
path: /metrics
port: metrics
# initialDelaySeconds: 0
# periodSeconds: 10
# timeoutSeconds: 1
# successThreshold: 1
failureThreshold: 30
# Configure the readinessProbe
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
readinessProbe:
httpGet:
path: /metrics
port: metrics
# initialDelaySeconds: 0
# periodSeconds: 10
# timeoutSeconds: 1
# successThreshold: 1
failureThreshold: 30
01_sources.conf
file definition in the fileConfigs part (leaving existing sources unaltered):## Fluentd configurations:
##
fileConfigs:
01_sources.conf: |-
<source>
@type udp
port 5140
bind 0.0.0.0
<parse>
@type json
</parse>
tag system
</source>
<match system.**>
@type relabel
@label @OUTPUT
</match>
<source>
@type tail
@id in_tail_container_logs
@label @KUBERNETES
02_filters.conf
section and add the block shown below for the Kubernetes.var.log.containers section. This part will handle JSON formatted logs read from the platform logs files (remember that this is not the MZ systemlog but files like platform log). 02_filters.conf: |-
<label @KUBERNETES>
<match kubernetes.var.log.containers.fluentd**>
@type relabel
@label @FLUENT_LOG
</match>
# <match kubernetes.var.log.containers.**_kube-system_**>
# @type null
# @id ignore_kube_system_logs
# </match>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
skip_labels false
skip_container_metadata false
skip_namespace_metadata true
skip_master_url true
</filter>
<filter kubernetes.var.log.containers.**>
@type parser
<parse>
@type json
json_parser json
</parse>
replace_invalid_sequence true
emit_invalid_record_to_error false
key_name log
reserve_data true
remove_key_name_field true
</filter>
<match **>
@type relabel
@label @DISPATCH
</match>
04_outputs.conf
section as below. This hostname is must match the hostname of the elasticsearch pod running in Kubernetes. The scheme must be HTTPS protocol as HTTP is not supported anymore. The password must be specified here and also the location of the certificates (the certificates are in the /elasticsearch
directory because of the volume mount we defined earlier in the values.yaml). 04_outputs.conf: |-
<label @OUTPUT>
<match **>
@type elasticsearch
host "elasticsearch-master"
port 9200
path ""
user elastic
password password
scheme https
ca_file /elasticsearch/ca.crt
client_cert /elasticsearch/tls.crt
client_key /elasticsearch/tls.key
</match>
</label>
helm upgrade --install fluentd /home/mzadmin/pe/fluentd -n logging
Enable System Log forwarding over Syslog
Verify that all services are up and running
Kubernetes 1/1
) only after it has started and successfully connected to elasticsearch. If the connection fails, then the pod will stay in 0/1
status. This can be due to problems in creating the keys in elasticsearch or in the configuration of the connection from fluentd to elasticsearch (in 04_outputs.conf
section in fluentd values.yaml
file). For more information on the documentation of fluentd - elasticsearch plugin, refer to GitHub - uken/fluent-plugin-elasticsearch. $ kubectl get pod -n logging
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 43m
elasticsearch-master-1 1/1 Running 0 44m
elasticsearch-master-2 1/1 Running 0 44m
fluentd-8g95p 1/1 Running 0 19h
fluentd-sqv7j 1/1 Running 0 19h
fluentd-zgx6t 1/1 Running 0 19h
kibana-kibana-56c9f469d-l7dtv 1/1 Running 0 105m
kubectl get service -n logging kibana-kibana -o jsonpath={.status.loadBalancer.ingress[0].hostname}
"thread": "main",
"level": "WARN",
"loggerName": "com.digitalroute.picostart.PlatformClassLoader",
"marker": {
"name": "PS",
"parents": [
{
"name": "LIFECYCLE"
}
]
},
"message": "Starting Web Server",
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
"instant": {
"epochSecond": 1614690212,
"nanoOfSecond": 819721000
},
"threadId": 1,
"threadPriority": 5,
"timestamp": "2021-03-02T13:03:32.819+0000"
"log": "2021-03-02 13:03:35.788:INFO:oejs.Server:pool-10-thread-1: Started @33338ms\n",