Hello!
I have 3 containers of docker.elastic.co/beats/filebeat:7.6.1 in Nomad. They stop to send logs once a day in random time and begin to waste cpu iowait. Restart of container helps approximatelly for a day.
Configuration of filebeat is listed below:
Summary
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: false
labels.dedot: true
templates:
- condition:
not.contains:
docker.container.name: filebeat
config:
- type: container
paths:
- '/var/lib/docker/containers/${data.docker.container.id}/*.log'
enabled: true
exclude_lines: ['.*customline*']
combine_partial: true
json:
keys_under_root: false
message_key: log
processors:
- add_docker_metadata: ~
tail_files: true
setup:
template:
name: "filebeat-index"
pattern: "filebeat-index-*"
overwrite: false
settings.index.number_of_shards: 3
ilm.enabled: false
kibana:
host: "kibana-host"
output.elasticsearch:
hosts: ["elastic-host:9200"]
index: "filebeat-index-%{+yyyy.MM.dd}"
bulk_max_size: 1000
workers: 2
I have the same configuration on 2 other Nomad clusters with less log data and it works fine there.
But this problem cluster probably has too much containers. And I think the issue is in discovery.
Could you suggest me anything, please?