# Logstash sincedb file not getting updated in ELK 6.8.0

**URL:** <https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524>\
**Category:** Logstash\
**Created:** [July 2, 2019, 1:57pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524 "2019-07-02T13:57:10Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sureshrupnar](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@sureshrupnar](https://discuss.elastic.co/u/sureshrupnar)\
**Post date:** [July 2, 2019, 1:57pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/1 "2019-07-02T13:57:10Z")

</div>

Hi,

recently we have migrated to ELK 6.8.0 and while ingesting data via logstash we are using  
sincedb\_path =\> "/export/vendor.sincedb"

in input plugin inside the file component.

This file was updated regularly after certain interval while data ingestion process is running in earlier ELK 5.5.x.  
But since we moved to newer version of ELK 6.8.0, this feature is not working.

Due to this our further processing was depended on it and it is currently failing.

Can you please suggest here how logstash can update SinceDb file after certain interval.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2019, 2:09pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/2 "2019-07-02T14:09:34Z")

</div>

What does the file input configuration look like? If it isn't doing what you expect you could enable '--log.level trace' on the command line and see what filewatch is doing.

---

<div class="post-metadata">

**Author:** ![sureshrupnar](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@sureshrupnar](https://discuss.elastic.co/u/sureshrupnar)\
**Post date:** [July 3, 2019, 1:42pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/3 "2019-07-03T13:42:37Z")

</div>

Here is my file input configuration -  
file{  
path =\> "/uat/home/source/vendor/working/vendor.txt"  
type =\>"VENDOR\_DATA"  
start\_position =\> "beginning"  
sincedb\_path =\> "/uat/home/sincedb/vendor\_type.sincedb"  
sincedb\_write\_interval =\> 10  
}  
}

Please find below logs -

> [2019-07-03T14:19:14,814][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
> /uat/home/installation/elk/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/sincedb\_collection.rb:22:in `initialize' /uat/home/installation/elk/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/observing_base.rb:62:in `build\_watch\_and\_dependencies'  
> /uat/home/installation/elk/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.10/lib/filewatch/observing\_base.rb:56:in `initialize'  
> [2019-07-03T14:20:40,303][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
> [2019-07-03T14:31:09,712][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
> [2019-07-03T14:31:09,720][TRACE][filewatch.sincedbcollection] open: reading from /uat/home/sincedb/vendor\_type.sincedb  
> [2019-07-03T14:31:09,720][TRACE][filewatch.sincedbcollection] open: count of keys read: 0  
> [2019-07-03T14:31:09,724][TRACE][filewatch.discoverer] discover\_files {"count"=\>1}  
> [2019-07-03T14:31:09,730][TRACE][filewatch.discoverer] discover\_files handling: {"new discovery"=\>true, "watched\_file details"=\>"\<FileWatch::WatchedFile: @filename='vendor.txt', @state='watched', @recent\_states='[:watched]', @bytes\_read='0', @bytes\_unread='0', current\_size='719516801', last\_stat\_size='719516801', file\_open?='false', @initial=true, @sincedb\_key='1073743100 0 64782'\>"}  
> [2019-07-03T14:31:09,730][TRACE][filewatch.sincedbcollection] associate: finding {"inode"=\>"1073743100", "path"=\>"/uat/home/source/vendor/working/vendor.txt"}  
> [2019-07-03T14:31:09,731][TRACE][filewatch.sincedbcollection] associate: unmatched  
> [2019-07-03T14:31:09,738][TRACE][filewatch.tailmode.processor] Delayed Delete processing  
> [2019-07-03T14:31:09,738][TRACE][filewatch.tailmode.processor] Watched + Active restat processing  
> [2019-07-03T14:31:09,739][TRACE][filewatch.tailmode.processor] Rotation In Progress processing  
> [2019-07-03T14:31:09,739][TRACE][filewatch.tailmode.processor] Watched processing  
> [2019-07-03T14:31:09,739][TRACE][filewatch.tailmode.handlers.createinitial] handling: avox\_cftc.txt  
> [2019-07-03T14:31:09,740][TRACE][filewatch.tailmode.handlers.createinitial] opening vendor.txt  
> [2019-07-03T14:31:09,740][TRACE][filewatch.tailmode.handlers.createinitial] handle\_specifically opened file handle: 239, path: vendor.txt  
> [2019-07-03T14:31:09,741][TRACE][filewatch.tailmode.handlers.createinitial] add\_new\_value\_sincedb\_collection {"position"=\>0, "watched\_file details"=\>"\<FileWatch::WatchedFile: @filename=vendor.txt', @state='active', @recent\_states='[:watched, :watched]', @bytes\_read='0', @bytes\_unread='719516801', current\_size='719516801', last\_stat\_size='719516801', file\_open?='true', @initial=true, @sincedb\_key='1073743100 0 64782'\>"}  
> [2019-07-03T14:31:09,741][TRACE][filewatch.tailmode.processor] Active - file grew: vendor.txt: new size is 719516801, bytes read 0  
> [2019-07-03T14:31:09,742][TRACE][filewatch.tailmode.handlers.grow] handling: vendor.txt  
> [2019-07-03T14:31:09,742][TRACE][filewatch.tailmode.handlers.grow] reading... {"iterations"=\>21957, "amount"=\>32768, "filename"=\>"vendor.txt"}  
> [2019-07-03T14:31:09,742][DEBUG][filewatch.tailmode.handlers.grow] read\_to\_eof: get chunk  
> [2019-07-03T14:31:09,863][DEBUG][filewatch.tailmode.handlers.grow] read\_to\_eof: get chunk

---

<div class="post-metadata">

**Author:** ![sureshrupnar](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@sureshrupnar](https://discuss.elastic.co/u/sureshrupnar)\
**Post date:** [July 4, 2019, 1:14pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/4 "2019-07-04T13:14:07Z")

</div>

Currently I am quite stuck, can you check the input configuration and shared logs. let me know, if you need any further information

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 4, 2019, 1:21pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/5 "2019-07-04T13:21:17Z")

</div>

That suggests that there are no line terminators in the file. It knows it needs to read 719 MB, and it is trying to do so, it is just not finding any lines in the file.

---

<div class="post-metadata">

**Author:** ![sureshrupnar](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@sureshrupnar](https://discuss.elastic.co/u/sureshrupnar)\
**Post date:** [July 5, 2019, 5:28am UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/6 "2019-07-05T05:28:29Z")

</div>

But same file is perfectly with Logstash (ELK 5.5.x), and we are getting expected output in sincedb file. Also in these files, the line separator is already present.

When I use below command to check the number of lines in file -  
wc -l vendor.txt

Then I am getting value as "1834549". So that means file has line separator.

---

<div class="post-metadata">

**Author:** ![sureshrupnar](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@sureshrupnar](https://discuss.elastic.co/u/sureshrupnar)\
**Post date:** [July 9, 2019, 6:34am UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/7 "2019-07-09T06:34:56Z")

</div>

Hi,

Can we get more pointer to drill down this issue and find out the root cause of this issue.

---

<div class="post-metadata">

**Author:** ![sureshrupnar](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@sureshrupnar](https://discuss.elastic.co/u/sureshrupnar)\
**Post date:** [July 10, 2019, 6:48am UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/8 "2019-07-10T06:48:18Z")

</div>

Even i am able to ingest all records into elasticsearch. And all records are ok.

Only problem is with sincedb file is not updating and even file size of sincedb file is 0kb.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 7, 2019, 6:48am UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524/9 "2019-08-07T06:48:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.