Filebeat stopped harvesting logs

[Sai_Avinash_Duddupud (avinash)]

Hi all,

I have started filebeat today with the following config and filbeat read logs till 2days ago and now it stopped receiving logs. I have tried restarting the filebeat by deleted the registry folder too, but the result is same.

it just repeats saying harvester for file is still running

filebeat.yml

filebeat.inputs:
- type: log
  paths:
    - /your/path/goes/here/*.log
output.logstash:
  enabled: true
  hosts: ["localhost:5044"]
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Filebeat logs

|2020-07-12T19:52:57.765Z|DEBUG|[input]|log/input.go:421|Check file for harvesting: /opt/deployment/unityapp/unity/npm-debug.log|
|---|---|---|---|---|
|2020-07-12T19:52:57.765Z|DEBUG|[input]|log/input.go:511|Update existing file for harvesting: /opt/deployment/unityapp/unity/npm-debug.log, offset: 96972|
|2020-07-12T19:52:57.765Z|DEBUG|[input]|log/input.go:563|Harvester for file is still running: /opt/deployment/unityapp/unity/npm-debug.log|
|2020-07-12T19:52:57.765Z|DEBUG|[input]|log/input.go:212|input states cleaned up. Before: 2, After: 2, Pending: 0|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|input/input.go:141|Run input|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:191|Start next scan|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:421|Check file for harvesting: /opt/deployment/unityapp/unity/django_blend.log|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:511|Update existing file for harvesting: /opt/deployment/unityapp/unity/django_blend.log, offset: 222051|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:563|Harvester for file is still running: /opt/deployment/unityapp/unity/django_blend.log|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:421|Check file for harvesting: /opt/deployment/unityapp/unity/npm-debug.log|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:511|Update existing file for harvesting: /opt/deployment/unityapp/unity/npm-debug.log, offset: 96972|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:563|Harvester for file is still running: /opt/deployment/unityapp/unity/npm-debug.log|
|2020-07-12T19:53:07.765Z|DEBUG|[input]|log/input.go:212|input states cleaned up. Before: 2, After: 2, Pending: 0|
|2020-07-12T19:53:17.765Z|DEBUG|[input]|input/input.go:141|Run input|
|2020-07-12T19:53:17.765Z|DEBUG|[input]|log/input.go:191|Start next scan|
|2020-07-12T19:53:17.766Z|DEBUG|[input]|log/input.go:421|Check file for harvesting: /opt/deployment/unityapp/unity/django_blend.log|
|2020-07-12T19:53:17.766Z|DEBUG|[input]|log/input.go:511|Update existing file for harvesting: /opt/deployment/unityapp/unity/django_blend.log, offset: 222051|
|2020-07-12T19:53:17.766Z|DEBUG|[input]|log/input.go:563|Harvester for file is still running: /opt/deployment/unityapp/unity/django_blend.log|
|2020-07-12T19:53:17.766Z|DEBUG|[input]|log/input.go:421|Check file for harvesting: /opt/deployment/unityapp/unity/npm-debug.log|
|2020-07-12T19:53:17.766Z|DEBUG|[input]|log/input.go:511|Update existing file for harvesting: /opt/deployment/unityapp/unity/npm-debug.log, offset: 96972|
|2020-07-12T19:53:17.766Z|DEBUG|[input]|log/input.go:563|Harvester for file is still running: /opt/deployment/unityapp/unity/npm-debug.log|
|2020-07-12T19:53:17.766Z|DEBUG|[input]|log/input.go:212|input states cleaned up. Before: 2, After: 2, Pending: 0|

Please suggest how to troubleshoot filebeat inorder to continue receiving the logs from log file

[kumarabhi (Abhishek)]

For debugging purpose, please redirect the Filebeat output to stdout

output {
	stdout { 
	     codec => rubydebug { } 
	}
}

Also, look at Harvester closing options in https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html

Can you also check that some new data is being added to the input file ?

[Sai_Avinash_Duddupud (avinash)]

@kumarabhi , logstash is now receving logs in real time although i couldn't see the logs in filebeat logs to my surprise