High disk usage while docker container is recreating

Elastic Stack Beats
1

Hi there.
I have Filebeat container that configured to scan redis logs from redis container. There is configuration (filebeat.yml):

filebeat.registry.flush: 2s

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOSTS:elasticsearch:9200}']
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

filebeat.modules:
  - module: redis
    slowlog:
        enabled: false

setup.kibana:
    host: '${KIBANA_HOST:kibana:5601}'
    username: '${KIBANA_USER:}'
    password: '${KIBANA_PASSWORD:}'

setup.dashboards.enabled: true

filebeat.autodiscover:
    providers:
      - type: docker
        templates:
          - condition:
                contains:
                    docker.container.image: redis
            config:
              - module: redis
                log:
                    input:
                        type: container
                        close_inactive: 1m
                        scan_frequency: 10s
                        paths:
                          - /var/lib/docker/containers/${data.docker.container.id}/*.log

After I've started the containers and then I've tried to recreate redis container, I got 100% disk usage. How can I fix that?

2

Hey @homa_inc, welcome to discusss :slight_smile:

What host is getting 100% disk usage? The one running Filebeat, or the one running Elasticsearch?

If it is the one running Filebeat, could you check with lsof if Filebeat is keeping many files open?

Filebeat keeps files open while it reads them, this can prevent files to be completely removed from disk if they are removed or rotated. This is the expected default behaviour to avoid losing logs that are rotated too quickly.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.