High disk usage while docker container is recreating

[homa_inc (Philipp Khamitsevich)]

Hi there.
I have Filebeat container that configured to scan redis logs from redis container. There is configuration (filebeat.yml):

filebeat.registry.flush: 2s

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOSTS:elasticsearch:9200}']
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

filebeat.modules:
  - module: redis
    slowlog:
        enabled: false

setup.kibana:
    host: '${KIBANA_HOST:kibana:5601}'
    username: '${KIBANA_USER:}'
    password: '${KIBANA_PASSWORD:}'

setup.dashboards.enabled: true

filebeat.autodiscover:
    providers:
      - type: docker
        templates:
          - condition:
                contains:
                    docker.container.image: redis
            config:
              - module: redis
                log:
                    input:
                        type: container
                        close_inactive: 1m
                        scan_frequency: 10s
                        paths:
                          - /var/lib/docker/containers/${data.docker.container.id}/*.log

After I've started the containers and then I've tried to recreate redis container, I got 100% disk usage. How can I fix that?

[jsoriano (Jaime Soriano)]

Hey @homa_inc, welcome to discusss

What host is getting 100% disk usage? The one running Filebeat, or the one running Elasticsearch?

If it is the one running Filebeat, could you check with lsof if Filebeat is keeping many files open?

Filebeat keeps files open while it reads them, this can prevent files to be completely removed from disk if they are removed or rotated. This is the expected default behaviour to avoid losing logs that are rotated too quickly.