Filebeat with container discovery stucks once a day

Elastic Stack Beats
1

Hello!

I have 3 containers of docker.elastic.co/beats/filebeat:7.6.1 in Nomad. They stop to send logs once a day in random time and begin to waste cpu iowait. Restart of container helps approximatelly for a day.

Configuration of filebeat is listed below:

Summary 
filebeat.autodiscover:
  providers:
- type: docker
  hints.enabled: false
  labels.dedot: true
  templates:
    - condition:
        not.contains:
          docker.container.name: filebeat
      config:
       - type: container
         paths:
           - '/var/lib/docker/containers/${data.docker.container.id}/*.log'
         enabled: true
         exclude_lines: ['.*customline*']
         combine_partial: true
         json:
           keys_under_root: false
           message_key: log
         processors:
           - add_docker_metadata: ~
         tail_files: true
setup:
  template:
name: "filebeat-index"
pattern: "filebeat-index-*"
overwrite: false
settings.index.number_of_shards: 3
  ilm.enabled: false
  kibana:
host: "kibana-host"

output.elasticsearch:
  hosts: ["elastic-host:9200"]
  index: "filebeat-index-%{+yyyy.MM.dd}"
  bulk_max_size: 1000
  workers: 2

I have the same configuration on 2 other Nomad clusters with less log data and it works fine there.
But this problem cluster probably has too much containers. And I think the issue is in discovery.
Could you suggest me anything, please?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.