Hello Loggers,
I just wanted to share weird behavior I experienced in certain scenarios when using Logstash. The scenarios are the following:
Scenario 1
-
Logstash is stopped
-
I delete any existing sincedb files
-
I clear Logstash Log files </opt/logstash/loglogstash.*>
-
I set a Logstash file input with the setting of <start_position => "beginning">
-
I set a Logstash file input path pattern to a directory where there are a certain amount of Log files
-
I start Logstash
RESULT: No files processed. An empty sincedb file is created after stopping Logstash process. -
I repeat steps 1-6, but instead change the file input path pattern, and move these log files to a different directory accordingly
RESULT: All files are processed by Logstash. A sincedb files with details from only one file is made during the Logstash process. (If I put another Log file in the path directory, that file is also processed by Logstash and the details of that file are added to the sincedb file, and the missing details of the previous files that were not in the sincedb file are also added.
Scenario 2
- Logstash is stopped
- I delete any existing sincedb files
- I clear Logstash Log files </opt/logstash/loglogstash.*>
- I set a Logstash file input with the setting of <start_position => "beginning">
- I set a Logstash file input path pattern to a directory where there are NO log files
- I start Logstash
- I move a tarred file of many log files in the path pattern directory
- I untar the tar file in the path pattern directory
RESULT: No files processed.
An empty sincedb file is created after stopping Logstash process.
-Start Logstash again
Sincedb file was populated with "0 0 0" values after second Logstash stop.
-Start Logstash again
Sincedb file was populated with "0 0 0 0" values after third Logstash stop.
I'd also like to explain a comparison fo these two scenarios, with the exception of scenario 1 not having step 7. The symptoms are identical:
DIFFERENT SCENARIO BUT SAME SYMPTOMS
Scenario 1 (Untarring) (start_position => "beginning")
- Logstash is stopped
- I delete any existing sincedb files
- I clear Logstash Log files </opt/logstash/loglogstash.*>
- I set a Logstash file input with the setting of <start_position => "beginning">
- I set a Logstash file input path pattern to a directory where there are NO log files
- I start Logstash
- I move a tarred file of many log files in the path pattern directory
- I untar the tar file in the path pattern directory
RESULT: No files processed.
An empty sincedb file is created after stopping Logstash process.
-Start Logstash again
Sincedb file was populated with "0 0 0" values after second Logstash stop.
-Start Logstash again
Sincedb file was populated with "0 0 0 0" values after third Logstash stop.
Scenario 2 (Normal) (start_position => "beginning")
- Logstash is stopped
- I delete any existing sincedb files
- I clear Logstash Log files </opt/logstash/loglogstash.*>
- I set a Logstash file input with the setting of <start_position => "beginning">
- I set a Logstash file input path pattern to a directory where there are a certain amount of Log files
- I start Logstash
RESULT: No files processed.
An empty sincedb file is created after stopping Logstash process.
-Start Logstash again
Sincedb file was populated with "0 0 0" values after second Logstash stop.
-Start Logstash again
Sincedb file was populated with "0 0 0 0" values after third Logstash stop.
Update: I missed mentioning that when I was moving the file, I was moving the file to a different filesystem, so the inode of the file changed. I tried moving the file to a different directory within the same file system, and the file was not processed. I beleive that the sincedb file is not working properly, or that Logstash has a persistent cache somewhere that Logstash keeps track of the indodes besides the sincedb file.
Any explanations or redirections to where I can further understand the deeper mechanics of Logstash? Thanks in advance guys, I really do appreciate your time for considering this.