I'm using the file input on my mac, and every time I update the file, it's replaying all messages from the beginning of the log.
I've tried explicitly setting the sincedb_path and start_position vars, but no luck.
.sincedb is created, but I noticed the inode didn't match the file I was updating.
On further investigation, (and I could be wrong), it looks like OSX changes the inode every time the file is updated, not just when a new file is created.
I assume this is the reason it resends all as Logstash thinks it's a new file, but wondering if anyone else has seen this, and/or has a workaround.
Indeed, it sounds like the file is rewritten in a new file. Logstash assumes that log files are appended to. What program is updating the file and is this an actual use case or just you testing?
vi probably creates a new file and renaming it to replace the original file, thus resulting in a new inode for the file every time. Make sure you append to the file instead, e.g. with echo "test log message" >> logfile.log.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
so it's not a big issue, but it's nicer to do my development on a Mac so would be good if it could be resolved.