Configure Log Collection, Target, and Visualization - AWS(3.0)
Note!
Some of the configuration requires the kubectl command. If this is your first time performing an AWSÂ Setup (3.0) to create the EKS cluster first.
installation, in order to use kubectl, you will have to set up the EKS cluster. Refer toBy default To get a production ready log configuration you can use tools from the Kubernetes ecosystem. In this guide we show you how to set up: These tools give you powerful and flexible log collection, storage, and visualization. The Elasticsearch database storage also provides powerful tools to perform analytics on the log data. It is outside the scope of this guide to describe those capabilities. Note! This guide includes the steps for installing the tools in the Kubernetes cluster. If you deploy Before setting up log collection, make sure that JSON formatted logging is enabled in in values.yaml. To isolate the logging functionality from the rest of the system and make it possible to reuse it from multiple It is recommended to enable Elasticsearch security before proceeding with installation: Note! For simplicity this example installs Elasticsearch without persistent storage. Refer to Elasticsearch Helm chart documentation for help to enable persistent storage: Edit values.yaml to enable System Log collection over syslog. Add the following source to section 01_sources.conf (leaving existing sources unaltered): Edit values.yaml to specify the JSON based log format: Configure according to: The parameters are set in Installation Instructions(3.0). If all looks good, you can get the URL for the Kibana dashboard with the below command: Open the Kibana dashboard and create a new Index Pattern matching the "fluentd" index. Go to Discover view to search collected log data. For instance, to search the platform log file, enter the search query "kubernetes.pod_name:platform" in the KQL field. Log records that are properly JSON formatted will be parsed into fields, like: While records that are not JSON formatted will be displayed per row in the "log" field, like:Enable JSON Logging in
log:
# Format can be "json" or "raw". Default is "raw"
format: json
Create a Namespace
kubectl create namespace logging
Add Helm Repos
helm repo add elastic https://helm.elastic.co
helm repo add fluent https://fluent.github.io/helm-charts
Install Elasticsearch
helm fetch elastic/elasticsearch --untar
https://github.com/elastic/helm-charts/tree/master/elasticsearchhelm upgrade --install elasticsearch elasticsearch -n logging --set=persistence.enabled=false
Install Kibana
helm fetch elastic/kibana --untar
helm upgrade --install kibana kibana -n logging --set=service.type=LoadBalancer --set=service.port=80
Install Fluentd
helm fetch fluent/fluentd --untar
01_sources.conf: |-
<source>
@type syslog
port 5140
bind 0.0.0.0
tag system
</source>
02_filters.conf: |-
<label @KUBERNETES>
<match kubernetes.var.log.containers.fluentd**>
@type relabel
@label @FLUENT_LOG
</match>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
skip_labels false
skip_container_metadata false
skip_namespace_metadata true
skip_master_url true
</filter>
<filter kubernetes.var.log.containers.**>
@type parser
<parse>
@type json
json_parser json
</parse>
replace_invalid_sequence true
emit_invalid_record_to_error false
key_name log
reserve_data true
remove_key_name_field true
</filter>
<match **>
@type relabel
@label @DISPATCH
</match>
</label>
helm upgrade --install fluentd fluentd -n logging
Enable System Log forwarding over Syslog
Verify that all services are up and running
$ kubectl get pod -n logging
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 43m
elasticsearch-master-1 1/1 Running 0 44m
elasticsearch-master-2 1/1 Running 0 44m
fluentd-8g95p 1/1 Running 0 19h
fluentd-sqv7j 1/1 Running 0 19h
fluentd-zgx6t 1/1 Running 0 19h
kibana-kibana-56c9f469d-l7dtv 1/1 Running 0 105m
kubectl get service -n logging kibana-kibana -o jsonpath={.status.loadBalancer.ingress[0].hostname}
"thread": "main",
"level": "WARN",
"loggerName": "com.digitalroute.picostart.PlatformClassLoader",
"marker": {
"name": "PS",
"parents": [
{
"name": "LIFECYCLE"
}
]
},
"message": "Starting Web Server",
"endOfBatch": false,
"loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
"instant": {
"epochSecond": 1614690212,
"nanoOfSecond": 819721000
},
"threadId": 1,
"threadPriority": 5,
"timestamp": "2021-03-02T13:03:32.819+0000"
"log": "2021-03-02 13:03:35.788:INFO:oejs.Server:pool-10-thread-1: Started @33338ms\n",