Sincedb doesn't updated until tail reaches the end of the file


it seems sincedb updating does not occur until TAIL reaches the end of the file.

with this configuration

input {
    file {
            path => "/home1/.logstash/json_input/deco.json.*"
            codec => "json"
            start_position => "beginning"
            add_field => ["[@metadata][input_id]", "deco_json"]
            sincedb_path => "/home1/.logstash/sincedb/deco_json.db2"

I daily move yesterday's log file into "/home1/.logstash/json_input/deco.json.YYYY.MM.DD"
It's size is approximately 1GB. and usually take 20~30minutes to process all line of the file.

when after yesterday's log file moved into logstash input directory, it is well tailed and out processed to elasticsearch.

one difficult thing I have with this situation,
If logstash or elasticsearch got break while process some line of the large file,
logstash have no sincedb position data, because it didn't reached EOF yet, sincedb position never wrote.
also this configuration have no effect.


in that case I don't know from which line should I re process the log data that didn't processed.
so, no way to failover.

any advice?
is this expected behavior?

I am using logstash 1.5.x


I think your observation is correct. The underlying filewatch library will read a file in a tight loop until it hits EOF and only afterwards will it consider updating sincedb. This seems like a bug to me—why not consider updating sincedb inside the read-32kb-and-yield loop? Could you please file an issue for the bug at Here's the relevant code:

thank you for the interest.

Okky thnks. I read few posts and added the sincedb path

        sincedb_path => "/persistent/log"        
         sincedb_write_interval => 10

Restarted my logstash started loading data as seen in the attachment..however again it gets stuck...and elasticsearch doesn't have data