Filebeat has started to have gaps in sending logs from a file written by rsyslog from a docker instance. Below the visualisation of count of documents from specific host
The path logs take to Kibana is:
Docker-> rsyslog -> file -> Filebeat -> Kafka -> Logstash -> Elasticsearch
Some notes:
- There has been no changes in the configuration of any of these.
- The decline of the logs is curious and is not a normal day distribution
- When looking at the documents ingested I can see the logs are showing up at similar time with when the log file was rotated. E.g.
-rw-------. 1 root root 478180336 Aug 27 18:04 /var/log/docker-containers.log
-rw-------. 1 root root 531935644 Aug 27 17:53 /var/log/docker-containers.log.1
-rw-------. 1 root root 538376090 Aug 27 17:40 /var/log/docker-containers.log.2
-rw-------. 1 root root 555042900 Aug 27 17:27 /var/log/docker-containers.log.3
-rw-------. 1 root root 534362012 Aug 27 17:15 /var/log/docker-containers.log.4
The amount of logs that come in at that time tho doesn't count to the expected amount of logs.
- I have tried configuring the path with both just the path and with a wildcard at the end