# Logstash is reindexing lastline and newline **URL:** \ **Category:** Logstash\ **Created:** [April 16, 2019, 4:01pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120 "2019-04-16T16:01:36Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 16, 2019, 4:01pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/1 "2019-04-16T16:01:37Z")
Hello community. I am using logstash to parse a logfile and index to elastic I have seen that if I enter a new line in the file, logstash is indexing the newline but also the last indexed line Looks like a bug an it comes with release 6.5.1, 6.6.1 and 7.0.0 Can you help please? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 16, 2019, 4:21pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/2 "2019-04-16T16:21:12Z")
What does your configuration look like? ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 16, 2019, 4:39pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/3 "2019-04-16T16:39:52Z")
This is my logstash.config file input { file { path =\> "/testfile.txt" start\_position =\> "end" sincedb\_path =\> "/temp/sincedb\_txt" } } filter { grok { patterns\_dir =\> ["./patterns"] match =\> { "message" =\> "%{TIMESTAMP:ts} (.\*) %{MESSAGE:msg} %{COLLECTOR:ip}@%{PORT:port}" } } if "\_grokparsefailure" in [tags] { drop { } } } output { elasticsearch { hosts =\> ["[http://x.y.z.c](http://x.y.z.c)"] index =\> "myindex" } } I am testing just inserting a line in the testfile.txt and I get 2 indexed documents in the elastic ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 16, 2019, 5:01pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/4 "2019-04-16T17:01:14Z")
Are the two documents the same? ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 16, 2019, 5:03pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/5 "2019-04-16T17:03:33Z")
No. Last line and new Inserted line ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 16, 2019, 6:07pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/6 "2019-04-16T18:07:05Z")
I cannot think of a way to explain that. You will need to enable TRACE level logging for the filewatch components. Take a look at [this](https://discuss.elastic.co/t/logstash-6-5-0-stops-shipping-logs-to-es/175477/5?u=badger) post for instructions. ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 16, 2019, 6:41pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/7 "2019-04-16T18:41:33Z")
Ok. I will work on this on Thursday. Tomorrow I am not at work. I will let you know Thank you Luca ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 16, 2019, 8:58pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/8 "2019-04-16T20:58:23Z")
Can you reproduce this always or was this a one off? ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 16, 2019, 9:01pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/9 "2019-04-16T21:01:20Z")
Yes I can reproduce this ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 16, 2019, 9:08pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/10 "2019-04-16T21:08:31Z")
Hmmm. I wondered for a minute that the sincedb was positioned just before the last line - but that would be a one-off thing. I will try to recreate tomorrow. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 17, 2019, 1:52pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/11 "2019-04-17T13:52:57Z")
I can recreate this but only under conditions where this behaviour is correct. Config: ```auto input { file { path => ["/elastic/tmp/testing/confs/test-file-read-repeat.log"] sincedb_path => "/elastic/tmp/testing/confs/test-file-tail-end.sdb" start_position => "end" } } filter { } output { stdout { codec => rubydebug } } ``` 1. Create a file with a single line, say "apples". 2. Start Logstash 3. No events are created. 4. Add a new line, say "strawberries" 5. An event with "strawberries" as the message field is emitted. 6. Shutdown Logstash 7. The sincedb file is written with the position set to 20 (length of apples + LF + length of strawberries + LF). 8. Remove the last line ("strawberries") from the file. 9. Start Logstash 10. An event with "apples" is emitted. 11. Add a new line, say "strawberries" 12. An event with "strawberries" as the message field is emitted. Please confirm that this is the recreation procedure you followed. ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 17, 2019, 2:57pm UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/12 "2019-04-17T14:57:17Z")
Tomorrow I can explain my procedure but it is not like this. Today I am not at work. I only add a line in a file but I get two documents indexed last line and new line. I don't remove any line from the file. I don't restart logstash. Regards Luca ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 18, 2019, 8:13am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/13 "2019-04-18T08:13:17Z")
Hello. **I understood what happens.** I have a file that is watched and is /test.txt Inside this file I have stuff and last line is document20. Logstash is closed. 1. I start logstash and nothing gets indexed. (correct) 2. I open the file /test.txt and add a line document21 3. I get indexed document 21 and document20 (not correct) 4. I put a new line document 22 with the command "echo document22 \>\> /test.txt" 5. I get indexed only document22 **(correct)** The problem is the way I put the entry in a file, opening it or using the echo command. In the fist way **I think** I create problem with the sincedb\_txt file. Sorry for the wrong procedure I used Luca ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 18, 2019, 8:26am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/14 "2019-04-18T08:26:32Z")
> [@anon73256657](#): > > 1. I start logstash and nothing gets indexed. (correct) Is this with an empty or missing sincedb file? ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 18, 2019, 8:57am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/15 "2019-04-18T08:57:49Z")
This is the sincedb file. 98307 0 64768 **108** 1555574697.254953 /test.txt 98309 0 64768 86 1555574632.993001 /test.txt 108 is the number length /test.txt is in bytes **I start logstash and get indexed nothing** ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 18, 2019, 9:00am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/16 "2019-04-18T09:00:54Z")
IIUC you are deleting `/test.txt` and recreating it - there are two inodes (records) in the sincedb file data. ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 18, 2019, 9:02am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/17 "2019-04-18T09:02:31Z")
is this because I entered the file in vi ? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 18, 2019, 9:05am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/18 "2019-04-18T09:05:14Z")
Probably. I don't know for sure. vi might not edit the original (i do recall from years back now this problem). ---
**Author:** ![anon73256657](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@anon73256657](https://discuss.elastic.co/u/anon73256657)\ **Post date:** [April 18, 2019, 9:07am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/19 "2019-04-18T09:07:35Z")
Any way if I put an entry in the file with echo command I get everything work **perfectly**. I am sorry I didnot expect vi could damage the sincedb Best regards Luca Razzi ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 18, 2019, 9:10am UTC](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120/20 "2019-04-18T09:10:57Z")
No worries. The sincedb is not really damaged, it is reporting the truth exactly. I go into a lot of detail about the sincedb and how file read positions are tracked in [this Discuss thread](https://discuss.elastic.co/t/logstash-7-w7-issue-input-file-issue-with-sincedb-reads-file-only-once/176571). Read my posts if you want a better understanding of a complex topic that seems simple on the surface. Good Luck. [Next page](https://discuss.elastic.co/t/logstash-is-reindexing-lastline-and-newline/177120.md?page=2)