currently i have a data.log file from which log stash is reading data continuously, after some one lakh lines i am moving this data to data1.log and adding some 100 lines of data into new data.log file. How does log stash is recognizing this change in data.log file and reading this new 100 lines of data. As byte offset in sincedb file corresponding to data.log will be pointing to some bigger number corresponding to old one lakh lines. So how does log stash is recognizing file rotation and resetting byte offset in since db files of that file.
currently i have a data.log file from which log stash is reading data continuously, after some one lakh lines
Please keep in mind that "lakh" is not a well-known term outside of India and neighboring countries.
i am moving this data to data1.log and adding some 100 lines of data into new data.log file. How does log stash is recognizing this change in data.log file and reading this new 100 lines of data. As byte offset in sincedb file corresponding to data.log will be pointing to some bigger number corresponding to old one lakh lines. So how does log stash is recognizing file rotation and resetting byte offset in since db files of that file.
Logstash tracks file offsets via inode numbers and will recognize that the original path no longer points to the same inode. It'll read the old file one last time (until EOF), forget that file, and reopen the new file to start tracking its contents.
I'm not sure if it'll read the new file from the beginning (which it should) or if it'll pay attention to the start_position option. In the latter case there would be a race condition where messages could be lost.
Thanks.