Logging and Monitoring EKS
Working Nodes Logging
- System logs from
- Application logs from application containers
- If instance is terminated, the logs are lost.
- Logs need to be aggregated in a meaningful way.
We set up a logging architecture to abstract logs from containers.
- Containerized app writes to
- System logs go to
- Container redirect logs to
We can add in a logging agent running as a DaemonSet to read logs and write to backend.
Finally, it is worth knowing of the EFK stack in Kubernetes:
- Amazon Elasticsearch Service
Fluentd vs Fluentbit
- fluentd has 100+ plugins, fluentbit has ~20 (2020).
As traffic goes up, fluentd can't keep up:
- fluentd based on Ruby and memory intensive
- slow propagation of logs
- loss of logs
- fluentd buffer can be increased to solve this but not dynamic
Fluentbit is lightweight and keeys up with higher traffic.
- fluentd to Kinesis Data Firehose to Logging backend
- fluentbit to logging backend
- hard to replace fluentd because of plugin support if already existing in enterprise