FileBeat misses lines at the beginning of a new file

Elastic Stack Beats
1

Sometimes Filebeat (5.3) misses some lines at the beginning of a new file.
From debug filebeat logs, it looks like filebeat detects an Update existing file for harvesting but it is not an upated/renamed/moved file, just a new file.

I imagine the os (centos 6.8) create a new log file with an inode equals to an already harvested and removed log file. Is that possible ?

FileBeat.yml :

filebeat.prospectors:
- paths:
  - /logm/*/logs/*.log
  
  
output.logstash:
  enabled: true
  hosts: ["server-1:5044"]

logging:
  level: debug
  to_files: true
  to_syslog: false
  files:
    path: logs
    name: a=filebeat~c=logm~h=server-1.log
    keepfiles: 100
    rotateeverybytes: 104857600

filebeat.log :

2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=5~l=69905.log
2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=57~l=69905.log
2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=57~l=69905.log, offset: 1182750
2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=57~l=69905.log
2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log
2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log, offset: 20479
2017-04-14T13:42:27Z DBG Resuming harvesting of file: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log, offset: 20479
2017-04-14T13:42:27Z DBG Set previous offset for file: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log. Offset: 20479
2017-04-14T13:42:27Z DBG Setting offset for file: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log. Offset: 20479
2017-04-14T13:42:27Z DBG Publish: {
"@timestamp": "2017-04-14T13:42:26.681Z",
"beat": {
"hostname": "server-1.novalocal",
"name": "server-1.novalocal",
"version": "5.3.0"
},
"input_type": "log",
"message": "2017-04-14T13:41:09.434Z#00000000000000013328#00000000000001999200#QD.69ZyS9.nVd - - [03/Sep/2759:13:07:48 -0800] "GET /vskDmIv9H HTTP/1.0" 2124 6779",
"offset": 1999200,
"source": "/logm/test/logs/c=logm~a=test~h=server-1~f=52~l=69905.log",
"type": "log"
}
2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/elasticsearch-e1/logs/logm-e1.log
2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/elasticsearch-e1/logs/logm-e1.log, offset: 8768
2017-04-14T13:42:27Z DBG File didn't change: /logm/elasticsearch-e1/logs/logm-e1.log
2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=1~l=69905.log
2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=1~l=69905.log, offset: 10485750
2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=1~l=69905.log
2017-04-14T13:42:27Z DBG Check file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=28~l=69905.log
2017-04-14T13:42:27Z INFO Harvester started for file: /logm/test/logs/c=logm~a=test~h=server-1~f=66~l=69905.log
2017-04-14T13:42:27Z DBG Update existing file for harvesting: /logm/test/logs/c=logm~a=test~h=server-1~f=28~l=69905.log, offset: 5935050
2017-04-14T13:42:27Z DBG Harvester for file is still running: /logm/test/logs/c=logm~a=test~h=server-1~f=28~l=69905.log

2

It seems like the same inode which was used for a previous file which was deleted, is now used again. Normally this can be solved by using clean_removed: true. But this is enabled by default.

What is the log rotating algorithm you are using?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.