I'm using the file input on my mac, and every time I update the file, it's replaying all messages from the beginning of the log.
I've tried explicitly setting the sincedb_path and start_position vars, but no luck.
.sincedb is created, but I noticed the inode didn't match the file I was updating.
On further investigation, (and I could be wrong), it looks like OSX changes the inode every time the file is updated, not just when a new file is created.
I assume this is the reason it resends all as Logstash thinks it's a new file, but wondering if anyone else has seen this, and/or has a workaround.
Indeed, it sounds like the file is rewritten in a new file. Logstash assumes that log files are appended to. What program is updating the file and is this an actual use case or just you testing?
Hi, it's just me testing, so using vi to add a new line to the file.
We don't use Mac's in production so it's not a big issue, but it's nicer to do my development on a Mac so would be good if it could be resolved.
vi probably creates a new file and renaming it to replace the original file, thus resulting in a new inode for the file every time. Make sure you append to the file instead, e.g. with
echo "test log message" >> logfile.log.
Good point, didn't even occur to me that might be the issue - thanks for pointer.