Filebeat Stops Consuming

Hi,

I can see Filebeat stopped consuming some log files that are not done yet.
When using debug level, I can see 4 files being read (2 old ones and 2 new ones):

|-08T15:23:40.795Z|DEBUG|[input]|log/input.go:404|Check file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-06.21|
|---|---|---|---|---|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:494|Update existing file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-06.21, offset: 19101842306|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:546|Harvester for file is still running: /ephemeral/log/haproxy/haproxy.2019-04-06.21|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:404|Check file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-07.00|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:494|Update existing file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-07.00, offset: 11099835919|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:546|Harvester for file is still running: /ephemeral/log/haproxy/haproxy.2019-04-07.00|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:404|Check file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-08.14|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:494|Update existing file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-08.14, offset: 16882069393|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:548|File didn't change: /ephemeral/log/haproxy/haproxy.2019-04-08.14|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:404|Check file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-08.15|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:494|Update existing file for harvesting: /ephemeral/log/haproxy/haproxy.2019-04-08.15, offset: 6760782677|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:546|Harvester for file is still running: /ephemeral/log/haproxy/haproxy.2019-04-08.15|
|2019-04-08T15:23:40.795Z|DEBUG|[input]|log/input.go:195|input states cleaned up. Before: 4, After: 4, Pending: 0|

However, looking at the logs or stats endpoint, it looks like there are 3 harvesters:

    "harvester": {
      "closed": 0,
      "open_files": 3,
      "running": 3,
      "skipped": 0,
      "started": 3
    },

The number of harvsters does not match the number of files in the directory. Moreover, only 2 files are actually consumed, not 4.

The file ends with an invalid JSON which causes Filebeat to not be able to reach the end of the file. When it doesn't reach the end, we can't delete that file (we're deleting files that were not touched for a while and that Filebeat got to the end of them).

How can I debug?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.