# Sincedb doesn't updated until tail reaches the end of the file

**URL:** <https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116>\
**Category:** Logstash\
**Created:** [December 2, 2015, 3:22am UTC](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116 "2015-12-02T03:22:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![no\_jihun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/no_jihun/32/64028_2.png) [@no\_jihun](https://discuss.elastic.co/u/no_jihun)\
**Post date:** [December 2, 2015, 3:22am UTC](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116/1 "2015-12-02T03:22:57Z")

</div>

hello.

it seems sincedb updating does not occur until TAIL reaches the end of the file.

with this configuration

```
input {
    file {
            path => "/home1/.logstash/json_input/deco.json.*"
            codec => "json"
            start_position => "beginning"
            add_field => ["[@metadata][input_id]", "deco_json"]
            sincedb_path => "/home1/.logstash/sincedb/deco_json.db2"
    }

```

I daily move yesterday's log file into "/home1/.logstash/json\_input/deco.json.YYYY.MM.DD"  
It's size is approximately 1GB. and usually take 20~30minutes to process all line of the file.

when after yesterday's log file moved into logstash input directory, it is well tailed and out processed to elasticsearch.

one difficult thing I have with this situation,  
If logstash or elasticsearch got break while process some line of the large file,  
logstash have no sincedb position data, because it didn't reached EOF yet, sincedb position never wrote.  
also this configuration have no effect.

```
 sincedb_write_interval

```

in that case I don't know from which line should I re process the log data that didn't processed.  
so, no way to failover.

any advice?  
is this expected behavior?

I am using logstash 1.5.x

thanks!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 2, 2015, 6:51am UTC](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116/2 "2015-12-02T06:51:32Z")

</div>

I think your observation is correct. The underlying filewatch library will read a file in a tight loop until it hits EOF and only afterwards will it consider updating sincedb. This seems like a bug to me—why not consider updating sincedb inside the read-32kb-and-yield loop? Could you please file an issue for the bug at [https://github.com/logstash-plugins/logstash-input-file/issues/new](https://github.com/logstash-plugins/logstash-input-file/issues/new)? Here's the relevant code:

> <https://github.com/jordansissel/ruby-filewatch/blob/v0.6.5/lib/filewatch/tail.rb#L178-L199>

---

<div class="post-metadata">

**Author:** ![no\_jihun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/no_jihun/32/64028_2.png) [@no\_jihun](https://discuss.elastic.co/u/no_jihun)\
**Post date:** [December 2, 2015, 7:00am UTC](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116/3 "2015-12-02T07:00:23Z")

</div>

> <https://github.com/logstash-plugins/logstash-input-file/issues/78>

thank you for the interest.

---

<div class="post-metadata">

**Author:** ![karthikvalluri85](https://avatars.discourse-cdn.com/v4/letter/k/a88e57/32.png) [@karthikvalluri85](https://discuss.elastic.co/u/karthikvalluri85)\
**Post date:** [September 9, 2016, 10:30am UTC](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116/4 "2016-09-09T10:30:35Z")

</div>

Okky thnks. I read few posts and added the sincedb path

```
        sincedb_path => "/persistent/log"        
         sincedb_write_interval => 10

```

Restarted my logstash ..it started loading data as seen in the attachment..however again it gets stuck...and elasticsearch doesn't have data

 ![](https://us1.discourse-cdn.com/elastic/original/2X/d/dbd0a3451d6e2f5a6d89872271f673e88ccc9fa0.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:39am UTC](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116/5 "2017-07-06T04:39:08Z")

</div>