# Closing file after its contents have been read

**URL:** <https://discuss.elastic.co/t/closing-file-after-its-contents-have-been-read/368696>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 11, 2024, 4:12pm UTC](https://discuss.elastic.co/t/closing-file-after-its-contents-have-been-read/368696 "2024-10-11T16:12:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ncluff](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ncluff/32/130399_2.png) [@ncluff](https://discuss.elastic.co/u/ncluff)\
**Post date:** [October 11, 2024, 4:12pm UTC](https://discuss.elastic.co/t/closing-file-after-its-contents-have-been-read/368696/1 "2024-10-11T16:12:56Z")

</div>

I'm using Filebeat 8.14.3 to collect lots of different files and the `close.reader.on_eof` doesn't seem to be working. The file only needs to be read once and then closed. If you need more of the filebeat.yml, please let me know.

I am sure its something I did, but I can't figure it out.

Filestream example in filebeat.yml

```auto
  - type: filestream
    enabled: true
    id: commands
    paths:
      - /usr/share/filebeat/sos/*/*/sos_commands/date/*
      - /usr/share/filebeat/sos/*/*/sos_commands/foobar_os/foobarcmd/*
      - /usr/share/filebeat/sos/*/*/sos_commands/foobar_os/user config/config_current.txt
      - /usr/share/filebeat/sos/*/*/sos_commands/hardware/dmidecode
      - /usr/share/filebeat/sos/*/*/sos_commands/logrotate/logrotate_debug
    parsers:
      - multiline:
          pattern: '^{.*}$' 
          negate: true
          match: after
    processors:
      - add_fields:
          fields:
            dest: commands
    close.reader.on_eof: true

```

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [October 11, 2024, 6:14pm UTC](https://discuss.elastic.co/t/closing-file-after-its-contents-have-been-read/368696/2 "2024-10-11T18:14:44Z")

</div>

> [@ncluff](#):
>
> and the `close.reader.on_eof` doesn't seem to be working

Can you share how you're testing this? What are you seeing which implies that read.on\_eof isnt working?

---

<div class="post-metadata">

**Author:** ![ncluff](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ncluff/32/130399_2.png) [@ncluff](https://discuss.elastic.co/u/ncluff)\
**Post date:** [October 11, 2024, 6:28pm UTC](https://discuss.elastic.co/t/closing-file-after-its-contents-have-been-read/368696/3 "2024-10-11T18:28:51Z")

</div>

Here is a little background. Basically I am getting a tar that I extract into a directory that Filebeat container(Docker) has access to and is monitoring those directories. I have other filestreams in the same filebeat.yml monitoring it as well.

When the tar is extracted I see the memory for filebeat go from MB to GB and it doesn't go down. Eventually the docker service will crash because too many files are open. As a test I would delete files that were extracted to see if Filebeats memory goes down and it did.

I can run Filebeat in debug to find better proof, but I don't know what selectors to choose. Would it be harvester?

Do I have `close.reader.on_eof: true` on the correct level in the yaml?

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [October 11, 2024, 6:55pm UTC](https://discuss.elastic.co/t/closing-file-after-its-contents-have-been-read/368696/4 "2024-10-11T18:55:26Z")

</div>

If you are dealing with a very large number of files I would try:

1. Limiting the number of harvesters via `harvester_limit`, perhaps to 25-100 or so from the default of unlimited
2. Setting `prospector.scanner.check_interval` to something longer than 10s like 30s to reduce how often the files are polled

That setting does look like it's set in the right place.

---

<div class="post-metadata">

**Author:** ![ncluff](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ncluff/32/130399_2.png) [@ncluff](https://discuss.elastic.co/u/ncluff)\
**Post date:** [October 11, 2024, 7:21pm UTC](https://discuss.elastic.co/t/closing-file-after-its-contents-have-been-read/368696/5 "2024-10-11T19:21:28Z")

</div>

Thank you for the direction. I will give it a go 🤞