# Can we change sincedb different location for different input? **URL:** \ **Category:** Logstash\ **Created:** [April 6, 2018, 4:53am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022 "2018-04-06T04:53:53Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 6, 2018, 4:53am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/1 "2018-04-06T04:53:53Z")
I have changed path.data on logstash.yml ``` path.data:D:/www/logstash-6.2.3/pr/ ``` Input file: ``` input { file { path => "D:/ELK-Sample-CSV/txt-5/test-import.csv" <-- this file have 100k lines start_position => "beginning" #sincedb_path => "D:/www/logstash-6.2.3/pr" sincedb_write_interval=> 1 } } filter { csv { separator => "," } } output { elasticsearch { hosts => "http://localhost:9200" index => "test-3" } stdout { codec => rubydebug } } ``` When I started running 1.Sincedb not created in mentioned location and also in default location 2. unexpectedly shut down, (imported 10k lines) 3. Restart again and I got this message ``` input { file { path => "D:/ELK-Sample-CSV/txt-5/test-import.csv" start_position => "end" <--- changed to end #sincedb_path => "D:/www/logstash-6.2.3/pr" sincedb_write_interval=> 1 } } filter { csv { separator => "," } } output { elasticsearch { hosts => "http://localhost:9200" index => "test-3" } stdout { codec => rubydebug } } ``` Now, I expected to read the line from where it left off? ``` The plugin keeps track of the current position in each file by recording it in a separate file named sincedb. This makes it possible to stop and restart Logstash and have it pick up where it left off without missing the lines that were added to the file while Logstash was stopped. Ex: file rotation is detect from 10001 line or else from unwatched line ``` Debug message: ``` _open_file: D:/ELK-Sample-CSV/txt-5/test-import.csv: opening D:/ELK-Sample-CSV/txt-5/test-import.csv: initial create, no sincedb, seeking to end 2007847 each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847 each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847 each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847 each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847 each: file grew: D:/ELK-Sample-CSV/txt-5/test-import.csv: old size 0, new size 2007847 Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#"} ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 6, 2018, 6:33am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/2 "2018-04-06T06:33:10Z")
You can configure the exact path to the sincedb file or indirectly set the directory (letting Logstash pick the filename) by changing the path.data setting. What you're trying to do doesn't work. If you explain what you're trying to accomplish maybe we can suggest something that solves that problem. ---
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 6, 2018, 10:19am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/4 "2018-04-06T10:19:11Z")
Please read again, I have updated my question. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 6, 2018, 11:29am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/5 "2018-04-06T11:29:19Z")
Perhaps no sincedb file was ever written? I wonder if the file perhaps is created only after the Logstash hits EOF on the input file, which it never did in this case. ---
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 6, 2018, 12:05pm UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/6 "2018-04-06T12:05:59Z")
Hi Magnus, Totally deleted logstash folder and reinstall, Run again same file, When I started running, above mentioned same procedures 1. Sincedb created in default location **D:\www\logstash-6.2.3\data\plugins\inputs\file** 2. unexpectedly shut down, (imported 10k lines) 3. Restart again and Mentioned Last created sincedb file Got the message: ``` [2018-04-06T17:33:07,836][DEBUG][logstash.inputs.file] _open_file: D:/ELK-Sample-CSV/txt-5/test-import.csv: opening [2018-04-06T17:33:07,867][DEBUG][logstash.inputs.file] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb last value 2007847, cur size 2007847 [2018-04-06T17:33:07,867][DEBUG][logstash.inputs.file] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb: seeking to 2007847 ``` I expected to import from where it **left off** , Can you understand my question? ---
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 6, 2018, 12:38pm UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/7 "2018-04-06T12:38:47Z")
Hi Magnus, Can update any sample code for unexpectedly shutdown cases, **example:** pipeline or system or server machine shutdown **when run the same input, How to resume from where it left?** , Because I using a big amount of data importing to elasticsearch on the daily basis. Logstash how to track or identify watched and unwatched files? I have read [Tracking of the current position in watched files](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files) and tried all the way, but no luck. How to achieve this? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 6, 2018, 1:12pm UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/8 "2018-04-06T13:12:31Z")
The log entries > [2018-04-06T17:33:07,867][DEBUG][logstash.inputs.file] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb last value 2007847, cur size 2007847 > [2018-04-06T17:33:07,867][DEBUG][logstash.inputs.file] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb: seeking to 2007847 indicate that the file's size is 2007847 bytes and that Logstash has reached that position, indicating that it actually _is_ continuing where it left off. ---
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 6, 2018, 1:24pm UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/9 "2018-04-06T13:24:37Z")
I understood from the log statement, but there is no new append to my existing index, Still, I have remaining data in my input file. Here I attached the log: ``` [2018-04-06T18:52:57,490][DEBUG][logstash.inputs.file] _open_file: D:/ELK-Sample-CSV/txt-5/test-import.csv: opening [2018-04-06T18:52:57,522][DEBUG][logstash.inputs.file] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb last value 2007847, cur size 2007847 [2018-04-06T18:52:57,526][DEBUG][logstash.inputs.file] D:/ELK-Sample-CSV/txt-5/test-import.csv: sincedb: seeking to 2007847 [2018-04-06T18:53:02,107][DEBUG][logstash.pipeline] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#"} [2018-04-06T18:53:02,474][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2018-04-06T18:53:02,536][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2018-04-06T18:53:07,116][DEBUG][logstash.pipeline] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#"} [2018-04-06T18:53:07,532][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2018-04-06T18:53:07,532][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2018-04-06T18:53:11,687][DEBUG][logstash.inputs.file] _globbed_files: D:/ELK-Sample-CSV/txt-5/test-import.csv: glob is: ["D:/ELK-Sample-CSV/txt-5/test-import.csv"] [2018-04-06T18:53:12,116][DEBUG][logstash.pipeline] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#"} [2018-04-06T18:53:12,544][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2018-04-06T18:53:12,544][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2018-04-06T18:53:17,124][DEBUG][logstash.pipeline] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#"} [2018-04-06T18:53:17,554][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} [2018-04-06T18:53:17,578][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"} [2018-04-06T18:53:22,125][DEBUG][logstash.pipeline] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#"} [2018-04-06T18:53:22,614][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"} ``` I don't see any error in the log, at the same time there are no new records in the existing index. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 6, 2018, 7:25pm UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/10 "2018-04-06T19:25:00Z")
Is the size of the file actually 2007847 bytes? ---
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 7, 2018, 3:59am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/11 "2018-04-07T03:59:57Z")
CSV file size is 2007847 bytes Sincedb value: ``` 385240477-968648-196608 0 0 2007847 ``` Is it possible to track unwatched line or inserting records from where it left at last run? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 10, 2018, 6:14pm UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/12 "2018-04-10T18:14:36Z")
Okay, but now I'm confused. Logstash clearly seeks to the end of the file so the problem doesn't appear to be that it doesn't pick up where it left off. What are you having problems with? ---
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 12, 2018, 5:10am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/13 "2018-04-12T05:10:18Z")
Actually, I'm based on Big Data, So, I have more than 50 million records in multiple CSV files, ``` path => "D:/ELK-Sample-CSV/pr/pr_*.csv" ``` **Logstash running from three different ec2 machines to import records.** Now, the problem is, 1. Imported 20 million records 2. Unfortunately shut down anyone server machine, 3. Again I have to run from the start\_position beginning or end? 4. How to rerun and import records from where it left? how to handle in this case? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 12, 2018, 6:19am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/14 "2018-04-12T06:19:47Z")
It might be the case that Logstash doesn't update the sincedb file until it has hit EOF, i.e. if you start reading a large file from scratch you won't be able to interrupt it, but if you continuously read a growing file it works fine. I believe Filebeat is much better at dealing with this. ---
**Author:** ![Karthik\_Saravanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthik_saravanan/32/28858_2.png) [@Karthik\_Saravanan](https://discuss.elastic.co/u/Karthik_Saravanan)\ **Post date:** [April 12, 2018, 6:57am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/15 "2018-04-12T06:57:06Z")
Thanks for your support @magnusbaeck, going to learn #beats ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 10, 2018, 6:57am UTC](https://discuss.elastic.co/t/can-we-change-sincedb-different-location-for-different-input/127022/16 "2018-05-10T06:57:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.