# Filebeats holding handles on rotated logs

**URL:** https://discuss.elastic.co/t/filebeats-holding-handles-on-rotated-logs/256538
**Category:** Beats
**Tags:** filebeat
**Created:** [November 24, 2020, 4:25pm UTC](https://discuss.elastic.co/t/filebeats-holding-handles-on-rotated-logs/256538 "2020-11-24T16:25:43Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![txmrlevine](https://avatars.discourse-cdn.com/v4/letter/t/5daacb/32.png) [@txmrlevine](https://discuss.elastic.co/u/txmrlevine)
#### Post date: [November 24, 2020, 4:25pm UTC](https://discuss.elastic.co/t/filebeats-holding-handles-on-rotated-logs/256538/1 "2020-11-24T16:25:44Z")

</div>

I am looking into why my disks are getting full and I found that filebeat is holding handles on logs that are rotated. I read some discussions on the topic and I did not see a definitive answer and all requests were rather old. I am running Filebeat 7.7.1 - I loaded lsof on a host to show what I am seeing. filebeats has been running for 2 weeks on this host and it has almost 3000 rotated logs that look like this.

```auto
filebeat 39160 113647 root 226r REG 253,2 10494498 263635 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat 39160 113647 root 227r REG 253,2 10485895 263636 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat 39160 113647 root 228r REG 253,2 10519906 263637 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat 39160 113647 root 229r REG 253,2 10488865 263638 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat 39160 113647 root 230r REG 253,2 10527631 263639 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat 39160 113647 root 231r REG 253,2 10486116 263640 /opt/active/logs/FoundationEmailService52.10.log (deleted)
filebeat 39160 113647 root 232r REG 253,2 10534540 263642 /opt/active/logs/FoundationEmailService52.10.log (deleted)

```

is there a setting somewhere that can fix this. here is my filebeat-yml it is very basic'

```auto
[root@prod-email-102w config]# cat filebeat-app.yml
filebeat.inputs:
- type: log
  enabled: true

  paths:
    - /opt/active/logs/*.log

  exclude_files: ['-gc\.log$','-tracer.*','\.\d+\.log$']

  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    match: after
    negate: True

filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.logstash:
  hosts: [SERVERS REMOVED FOR PROTECTION]

processors:
  - add_locale: ~

logging.level: info
logging.to_files: true
logging.files:
  path: /opt/filebeat/logs
  name: app.log
  keepfiles: 7
  permissions: 0644

```

---

<div class="post-metadata">

### Author: ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [November 24, 2020, 5:17pm UTC](https://discuss.elastic.co/t/filebeats-holding-handles-on-rotated-logs/256538/2 "2020-11-24T17:17:14Z")

</div>

Can you see any errors in the logs of Filebeat coming from the output? Or can you see any errors in the logs of Logstash?

What you are experiencing usually happens when the output, in this case Logstash, cannot keep up with the events coming from Filebeat.

---

<div class="post-metadata">

### Author: ![txmrlevine](https://avatars.discourse-cdn.com/v4/letter/t/5daacb/32.png) [@txmrlevine](https://discuss.elastic.co/u/txmrlevine)
#### Post date: [November 24, 2020, 6:01pm UTC](https://discuss.elastic.co/t/filebeats-holding-handles-on-rotated-logs/256538/3 "2020-11-24T18:01:40Z")

</div>

the only error I see is when it is loading modules it says it is not there.

cat app.log | grep ERROR  
2020-11-24T15:00:04.487Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /opt/filebeat/module

no issues in send.

here is my dirs

```auto
-sh-4.2$ cd /opt/filebeat/
-sh-4.2$ ls -lrta
total 24
drwxr-xr-x 9 root root 4096 Oct 1 15:23 ..
drwxr-xr-x 6 root root 4096 Oct 1 15:23 .
drwxr-xr-x 5 root root 4096 Oct 1 15:24 bin
drwxr-xr-x 4 root root 4096 Oct 1 15:27 data
drwxr-xr-x 3 root root 4096 Nov 10 15:04 config
drwxr-xr-x 2 root root 4096 Nov 24 15:00 logs
-sh-4.2$ cd config
-sh-4.2$ ls -lrta
total 20
drwxr-xr-x 2 root root 4096 Oct 1 15:23 inputs.d
drwxr-xr-x 6 root root 4096 Oct 1 15:23 ..
lrwxrwxrwx 1 root root 24 Oct 1 15:25 module -> /opt/filebeat/bin/module
lrwxrwxrwx 1 root root 27 Oct 1 15:25 modules.d -> /opt/filebeat/bin/modules.d
-rw-r--r-- 1 root root 775 Oct 1 15:25 filebeat-sys.yml
-rw-r--r-- 1 root root 795 Nov 10 15:04 filebeat-app.yml
drwxr-xr-x 3 root root 4096 Nov 10 15:04 .

```

so maybe I need to fix my config to point to the correct dir? I did not set this up an I am fairly new. we have pushed this agent to about 3K servers so any changes would need pushed. please let me know.

---

<div class="post-metadata">

### Author: ![txmrlevine](https://avatars.discourse-cdn.com/v4/letter/t/5daacb/32.png) [@txmrlevine](https://discuss.elastic.co/u/txmrlevine)
#### Post date: [December 1, 2020, 5:20pm UTC](https://discuss.elastic.co/t/filebeats-holding-handles-on-rotated-logs/256538/4 "2020-12-01T17:20:53Z")

</div>

I still am seeing the deleted files from handles I tried the ingore\_older setting and close\_inactive setting and neither does anything to help the situation. one of my hosts is very busy and rotates logs about every 5 minutes. Is there a way to stop this behavior and get filebeat to release the handles. I have issues with disk space filling because of this.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 29, 2020, 7:21pm UTC](https://discuss.elastic.co/t/filebeats-holding-handles-on-rotated-logs/256538/5 "2020-12-29T19:21:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
