in this post Filebeat and truncated files
@ruflin states that filebeat will detect when a file is truncated and start reading again.
I am on beats7.10 and centos 7
We are going to move all our logging to move and hup but for now I have some applications where we are using copytruncate
I see the following behaviour. After the log file is truncated I see a few lines in elasticsearch but then the data stops.
I looked in /var/lib/filebeat/registry/filebeat/log.json
I found the file in question. The inode is correct but the log offset is greater than the current filesize. So I wonder if filebeats got confused when the file was truncated and is waiting for the old log offset
config includes
scan_frequency: 10s
ignore_older: 24h
also paths does not match the rotated file
I see this in the debug log which seems ominous bit I have no idea why it is not recognising the inputs if that is the case.
2020-12-07T12:27:07.001Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0
2020-12-07T12:27:07.001Z DEBUG [registrar] registrar/registrar.go:140 Starting Registrar
2020-12-07T12:27:07.001Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2020-12-07T12:27:07.001Z INFO cfgfile/reload.go:164 Config reloader started
2020-12-07T12:27:07.001Z INFO cfgfile/reload.go:164 Config reloader started