Filebeats holding handles on rotated logs

[txmrlevine (Michael Levine)]

I am looking into why my disks are getting full and I found that filebeat is holding handles on logs that are rotated. I read some discussions on the topic and I did not see a definitive answer and all requests were rather old. I am running Filebeat 7.7.1 - I loaded lsof on a host to show what I am seeing. filebeats has been running for 2 weeks on this host and it has almost 3000 rotated logs that look like this.

filebeat   39160 113647     root  226r      REG              253,2  10494498      263635 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat   39160 113647     root  227r      REG              253,2  10485895      263636 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat   39160 113647     root  228r      REG              253,2  10519906      263637 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat   39160 113647     root  229r      REG              253,2  10488865      263638 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat   39160 113647     root  230r      REG              253,2  10527631      263639 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat   39160 113647     root  231r      REG              253,2  10486116      263640 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat   39160 113647     root  232r      REG              253,2  10534540      263642 /opt/active/logs/FoundationEmailService52.10.log (deleted)

is there a setting somewhere that can fix this. here is my filebeat-yml it is very basic'

[root@prod-email-102w config]# cat filebeat-app.yml
filebeat.inputs:
- type: log
  enabled: true

  paths:
    - /opt/active/logs/*.log

  exclude_files: ['-gc\.log$','-tracer.*','\.\d+\.log$']

  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    match: after
    negate: True


filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.logstash:
  hosts: [SERVERS REMOVED FOR PROTECTION ]

processors:
  - add_locale: ~

logging.level: info
logging.to_files: true
logging.files:
  path: /opt/filebeat/logs
  name: app.log
  keepfiles: 7
  permissions: 0644

[kvch (Noémi Ványi)]

Can you see any errors in the logs of Filebeat coming from the output? Or can you see any errors in the logs of Logstash?

What you are experiencing usually happens when the output, in this case Logstash, cannot keep up with the events coming from Filebeat.

[txmrlevine (Michael Levine)]

the only error I see is when it is loading modules it says it is not there.

cat app.log | grep ERROR
2020-11-24T15:00:04.487Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /opt/filebeat/module

no issues in send.

here is my dirs

-sh-4.2$ cd /opt/filebeat/
-sh-4.2$ ls -lrta
total 24
drwxr-xr-x 9 root root 4096 Oct  1 15:23 ..
drwxr-xr-x 6 root root 4096 Oct  1 15:23 .
drwxr-xr-x 5 root root 4096 Oct  1 15:24 bin
drwxr-xr-x 4 root root 4096 Oct  1 15:27 data
drwxr-xr-x 3 root root 4096 Nov 10 15:04 config
drwxr-xr-x 2 root root 4096 Nov 24 15:00 logs
-sh-4.2$ cd config
-sh-4.2$ ls -lrta
total 20
drwxr-xr-x 2 root root 4096 Oct  1 15:23 inputs.d
drwxr-xr-x 6 root root 4096 Oct  1 15:23 ..
lrwxrwxrwx 1 root root   24 Oct  1 15:25 module -> /opt/filebeat/bin/module
lrwxrwxrwx 1 root root   27 Oct  1 15:25 modules.d -> /opt/filebeat/bin/modules.d
-rw-r--r-- 1 root root  775 Oct  1 15:25 filebeat-sys.yml
-rw-r--r-- 1 root root  795 Nov 10 15:04 filebeat-app.yml
drwxr-xr-x 3 root root 4096 Nov 10 15:04 .

so maybe I need to fix my config to point to the correct dir? I did not set this up an I am fairly new. we have pushed this agent to about 3K servers so any changes would need pushed. please let me know.

[txmrlevine (Michael Levine)]

I still am seeing the deleted files from handles I tried the ingore_older setting and close_inactive setting and neither does anything to help the situation. one of my hosts is very busy and rotates logs about every 5 minutes. Is there a way to stop this behavior and get filebeat to release the handles. I have issues with disk space filling because of this.