File harvested but not sent

[lobando (Luis Obando)]

Hello,
This is my current setup.

1. Filebeat running in a docker container reading json files from a mounted docker volume.

2. These files are only read once and then close. close_eof = true

3. Files are written to the docker volume in bursts of many files per second.

4. Data sent to a hosted elastic cluster, using cloud.id option

5. I have a script that reads filebeat registry and then remove the file after is completely processed.

Problem:
Some files are never sent to elastic and they start accumulating in the docker volume, after doing some investigation, this is what's happening

1. Those files get completely harvested by filebeat, confirmed by looking at filebeat log

2. Files are no longer in filebeat registry (meaning again harvested done and sent confirmed?)

3. Like I said, file never show up in elastic and never gets deleted by custom script

4. Restarting filebeat fix the issue, file is harvested again and then sent.

Any idea of what could be happening ?

Current filebeat config

filebeat.inputs:
- type: log
  fields.type: 'stats'
  enabled: true
  close_eof: true
  json.keys_under_root: true
  paths:
    - /Logs/stats/*.stats.json
  harvester_limit: 150

This happen only to some files

[lobando (Luis Obando)]

any ideas here ? Thanks