When I started running
1.Sincedb not created in mentioned location and also in default location
2. unexpectedly shut down, (imported 10k lines)
3. Restart again and I got this message
Now, I expected to read the line from where it left off?
The plugin keeps track of the current position in each file by recording it in a separate file
named sincedb. This makes it possible to stop and restart Logstash and have it pick up
where it left off without missing the lines that were added to the file while Logstash was
stopped.
Ex: file rotation is detect from 10001 line or else from unwatched line
Debug message:
_open_file: D:/ELK-Sample-CSV/txt-5/test-import.csv: opening
D:/ELK-Sample-CSV/txt-5/test-import.csv: initial create, no sincedb, seeking to end 2007847
each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847
each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847
each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847
each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847
each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847
Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x5432d49b sleep>"}
You can configure the exact path to the sincedb file or indirectly set the directory (letting Logstash pick the filename) by changing the path.data setting. What you're trying to do doesn't work.
If you explain what you're trying to accomplish maybe we can suggest something that solves that problem.
Perhaps no sincedb file was ever written? I wonder if the file perhaps is created only after the Logstash hits EOF on the input file, which it never did in this case.
[2018-04-06T17:33:07,867][DEBUG][logstash.inputs.file ] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb last value 2007847, cur size 2007847
[2018-04-06T17:33:07,867][DEBUG][logstash.inputs.file ] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb: seeking to 2007847
indicate that the file's size is 2007847 bytes and that Logstash has reached that position, indicating that it actually is continuing where it left off.
I understood from the log statement, but there is no new append to my existing index, Still, I have remaining data in my input file.
Here I attached the log:
[2018-04-06T18:52:57,490][DEBUG][logstash.inputs.file ] _open_file: D:/ELK-Sample-CSV/txt-5/test-import.csv: opening
[2018-04-06T18:52:57,522][DEBUG][logstash.inputs.file ] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb last value 2007847, cur size 2007847
[2018-04-06T18:52:57,526][DEBUG][logstash.inputs.file ] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb: seeking to 2007847
[2018-04-06T18:53:02,107][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x32b6f385 sleep>"}
[2018-04-06T18:53:02,474][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-04-06T18:53:02,536][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-04-06T18:53:07,116][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x32b6f385 sleep>"}
[2018-04-06T18:53:07,532][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-04-06T18:53:07,532][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-04-06T18:53:11,687][DEBUG][logstash.inputs.file ] _globbed_files: D:/ELK-Sample-CSV/txt-5/test-import.csv: glob is: ["D:/ELK-Sample-CSV/txt-5/test-import.csv"]
[2018-04-06T18:53:12,116][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x32b6f385 sleep>"}
[2018-04-06T18:53:12,544][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-04-06T18:53:12,544][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-04-06T18:53:17,124][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x32b6f385 sleep>"}
[2018-04-06T18:53:17,554][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-04-06T18:53:17,578][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-04-06T18:53:22,125][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x32b6f385 sleep>"}
[2018-04-06T18:53:22,614][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
I don't see any error in the log, at the same time there are no new records in the existing index.
Okay, but now I'm confused. Logstash clearly seeks to the end of the file so the problem doesn't appear to be that it doesn't pick up where it left off. What are you having problems with?
It might be the case that Logstash doesn't update the sincedb file until it has hit EOF, i.e. if you start reading a large file from scratch you won't be able to interrupt it, but if you continuously read a growing file it works fine.
I believe Filebeat is much better at dealing with this.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.