# Logstash7.6 - data loss when input file logrotation happens

**URL:** <https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719>\
**Category:** Logstash\
**Created:** [July 3, 2020, 12:16am UTC](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719 "2020-07-03T00:16:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![manu1](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@manu1](https://discuss.elastic.co/u/manu1)\
**Post date:** [July 3, 2020, 12:16am UTC](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719/1 "2020-07-03T00:16:03Z")

</div>

Hello,

We have observed data loss when input file's logrotation happens every day.

```auto
    input {
      file {
        path => "/weblogs/biweb.log"
        sincedb_path => "/opt/logshipper/ls7_bi/sincedbs/biweb.db"
        type => "biweb.bi_event"
      }
    output {
      if [type] == "biweb.bi_event" {
        kafka {
          bootstrap_servers => 'KAFKA HOSTS'
          topic_id => "biweb.event"
          acks => "1"
          batch_size => 100
          linger_ms => 1000
          retry_backoff_ms => 5000
          retries => 10
          compression_type => "snappy"
          codec => plain { format => "%{message}" }

        }
      }
    }

```

This is the snippet of config.

Any suggestions on this issue will be helpful.

Thank you!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 3, 2020, 12:41am UTC](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719/2 "2020-07-03T00:41:22Z")

</div>

> [@manu1](#):
>
> We have observed data loss when input file's logrotation happens every day.

That could be an inode reuse issue. There are links to various issues in the META issue [211](https://github.com/logstash-plugins/logstash-input-file/issues/211). Especially see [251](https://github.com/logstash-plugins/logstash-input-file/issues/251).

Tracking which files have been read when those files can get rotated is an extremely hard problem. Way harder than most folks would initially think. A good option to get it right is to checksum the file contents (although this is not foolproof), and the file input does not do that, because it can get ridiculously expensive. Instead it implements a very cheap technique that almost always gets it right (but in a few cases it decides it has already read a file that it has not read).

There are other cases where it gets it wrong by [duplicating data](https://discuss.elastic.co/t/when-logstash-shutdown-renam-file-and-data-can-be-lost-or-duplicated/192669). As I said, it is a really hard problem.

---

<div class="post-metadata">

**Author:** ![manu1](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@manu1](https://discuss.elastic.co/u/manu1)\
**Post date:** [July 7, 2020, 9:00pm UTC](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719/3 "2020-07-07T21:00:56Z")

</div>

> [@Badger](#):
>
> 251

Thank you @Badger for the update.

We are using Logstash1.5 in some legacy pipelines and interestingly we haven't seen this data loss during logrotation in that pipeline.  
What is the main difference in Logstash1.5 vs Logstash7.6 to handle log rotation in input file plugin?

We have tried below options:

1. we have tried wildcard in **path** to handle the logrotation issue in LS 7.6, however it created huge number of duplicates(re-read all last 24 hour events) and we reverted back

```auto
        path => "/weblogs/biweb.log*"

```

1. We have tried tuning the pipeline parameters listed below, but no improvements on the data loss

```auto
pipeline.workers from 2 to 12 (default: 24) 
pipeline.batch.size from 125 to 250 (default: 125)

```

Currently _ **sincedb\_clean\_after** _ and _ **sincedb\_write\_interval** _ is not set, it is using default values(_sincedb\_clean\_after: 2 weeks, sincedb\_write\_interval : 15sec_). Do any of these property tuning will help?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 7, 2020, 10:09pm UTC](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719/4 "2020-07-07T22:09:28Z")

</div>

I do not think there are any good solutions.

The file input had a huge amount of work done on it between 1.5 and v5 (not so much recently). I could not summarize it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 4, 2020, 10:09pm UTC](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719/5 "2020-08-04T22:09:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.