...
We separate error conditions in 6 different layers as depicted below.
Infrastructure Layer
The infrastructure layer includes all AWS resources used by the application. These resources are typically controlled Infrastructure as Code as described in Assets and Services (2.3).
...
For the EKS cluster specifically, see https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html.
Orchestration Layer
General Kubernetes Troubleshooting
There are plenty of good guides on the internet to help you troubleshoot problems related to Kubernetes resources. Instead of describing such procedures here you are recommended to refer such guides. Some good examples are:
- Official Kubernetes documentation on Troubleshooting Applications: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/
- Official Kubernetes documentation on Troubleshooting Cluster https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/
- Troubleshooting guide from LearnK8s: https://learnk8s.io/troubleshooting-deployments
Kubernetes Resource Health Monitoring
All Kubernetes pods deployed by define Liveness and Readiness probes according to https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
...
These events often contain useful information for troubleshooting.
Kubernetes Logs
All processes in a system produce log data that is collected by Kubernetes log function and can easily be transferred to a common log target. Also the centralized System Log (System Log) can be forwarded to the same log target, The stack Fluentd, ElasticSearch and Kibana to collect, store and visualize log data. See Configure Log Collection, Target, and Visualization - AWS for a description on how to set this up.
If centralized logging is not used, all process logs can instead be accessed using the 'kubectl logs' command. For example:
System Layer
Event Notifications
To monitor error conditions in the system layer, has a very flexible Event Notifier feature with targets like AWS SNS Topic and several others. See Event Notifications (2.3) for info on how to configure this.
System and Process Logs
logs events from the entire system in the central System Log. See System Log (2.3)
Metrics
All metrics in are exposed on a REST interface in a format that can be scraped by Prometheus. This means if Prometheus is installed according to Setting up Prometheus (2.3) it will automatically start scraping metrics from all system resources.
Alerts
Since Prometheus is the central integration point for metrics it is straight forward to also implement alert functionality here using the Prometheus Alert Manager. See https://prometheus.io/docs/alerting/latest/alertmanager/ for details.
Verifying System Database Connectivity
If the system is not starting up and the platform process log indicates problems with attaching to the system database it's possible to check this connection by accessing Postgres command line tool through kubectl command.
|
Configuration Layer
Configuration Generation Errors
If for some a discrepancy in the dependencies between resources generated from workflows, Ultra definitions and other resources happen, the result will be error messages like "Corrupt type information" or similar can occur when trying to save or import resources. It might then be necessary to regenerate all resources to fix the inconsistencies. This is done my the regenerateconfigs command described in regenerateconfigs.
Execution Layer
Workflow Monitor
For workflow troubleshooting, the Workflow Monitor can be used to view detail debug information, see Workflow Monitor on the Web Interface (2.3).
Execution Layer Metrics
has extensive monitoring capabilities for troubleshooting workflow scheduling and execution. See Metrics (2.3) for more information on available metrics. See Reading JMX Metrics, MIMs and Prometheus Agent Metrics from Execution Context Endpoint (2.3) for information on how to expose execution layer the metrics in Prometheus and/or Grafana.
Troubleshooting EC Deployments
ECDeployments or ECDs are instances of a Kubernetes Custom Resource. Just like other Kubernetes resource an ECD has a status section that contain informative text describing the state of the resource. This information can be displayed using the kubectl command:
|
Data Layer
When unexpected things happen when processing payload data, like for instance decoding errors or unexpected type codes, provides a powerful subsystem called Data Veracity to help resolving the error condition. See Data Veracity (2.3) for information on how to configure this.