Filebeat not close delete filehandler

Elastic Stack Beats
,
1

we are running filebeat 6.8.22 with inputs as docker containers and it doesn't release file handlers for deleted files

filebeat.inputs:
- type: docker
  close_inactive: 5m
  close_removed: true
  clean_removed: true
  containers:
    ids: "*"
    path: /var/lib/docker/containers
    stream: all
  ignore_older: 36h
  json.message_key: log
  json.keys_under_root: false
  json.ignore_decoding_error: true
  scan_frequency: 60s
  document_type: 'docker-logs'
  processors:
  - add_docker_metadata: ~

an example from ps output

filebeat  27275            root    5r      REG              202,1 100000031  127955205 /var/lib/docker/containers/2839281c157
8e88f15249ba1cad623f48831e7b1972d7ecbb36b1dfc1a0fba69/2839281c1578e88f15249ba1cad623f48831e7b1972d7ecbb36b1dfc1a0fba69-json.l
og.2 (deleted)

lsof -n|grep filebeat|grep delete|wc -l
285

this behavior causes extra disk space on EC2 nodes to be used and not properly cleaned up, and eventually, used space is hitting 100% percent and causing Out of disks space errors, affecting other containers running at the same EC2 node

any ideas how to fix it? possible quick workaround can be periodic restart for filebeat container, but it seems hacky to me

GitHub issue - filebeat not close delete filehandler · Issue #30472 · elastic/beats · GitHub

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.