# File harvested but not sent

**URL:** <https://discuss.elastic.co/t/file-harvested-but-not-sent/197624>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 1, 2019, 5:24pm UTC](https://discuss.elastic.co/t/file-harvested-but-not-sent/197624 "2019-09-01T17:24:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lobando](https://avatars.discourse-cdn.com/v4/letter/l/b38774/32.png) [@lobando](https://discuss.elastic.co/u/lobando)\
**Post date:** [September 1, 2019, 5:24pm UTC](https://discuss.elastic.co/t/file-harvested-but-not-sent/197624/1 "2019-09-01T17:24:42Z")

</div>

Hello,  
This is my current setup.

1. Filebeat running in a docker container reading json files from a mounted docker volume.

2. These files are only read once and then close. `close_eof = true`

3. Files are written to the docker volume in bursts of many files per second.

4. Data sent to a hosted elastic cluster, using `cloud.id` option

5. I have a script that reads filebeat registry and then remove the file after is completely processed.

Problem:  
Some files are never sent to elastic and they start accumulating in the docker volume, after doing some investigation, this is what's happening

1. Those files get completely harvested by filebeat, confirmed by looking at filebeat log

2. Files are no longer in filebeat registry (meaning again harvested done and sent confirmed?)

3. Like I said, file never show up in elastic and never gets deleted by custom script

4. Restarting filebeat fix the issue, file is harvested again and then sent.

Any idea of what could be happening ?

Current filebeat config

```auto
filebeat.inputs:
- type: log
  fields.type: 'stats'
  enabled: true
  close_eof: true
  json.keys_under_root: true
  paths:
    - /Logs/stats/*.stats.json
  harvester_limit: 150

```

This happen only to some files

---

<div class="post-metadata">

**Author:** ![lobando](https://avatars.discourse-cdn.com/v4/letter/l/b38774/32.png) [@lobando](https://discuss.elastic.co/u/lobando)\
**Post date:** [September 9, 2019, 2:08pm UTC](https://discuss.elastic.co/t/file-harvested-but-not-sent/197624/2 "2019-09-09T14:08:19Z")

</div>

any ideas here ? Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 7, 2019, 2:08pm UTC](https://discuss.elastic.co/t/file-harvested-but-not-sent/197624/3 "2019-10-07T14:08:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.