# Logstash lost data when using file input plugin

**URL:** <https://discuss.elastic.co/t/logstash-lost-data-when-using-file-input-plugin/209406>\
**Category:** Logstash\
**Created:** [November 26, 2019, 3:28am UTC](https://discuss.elastic.co/t/logstash-lost-data-when-using-file-input-plugin/209406 "2019-11-26T03:28:45Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bigben](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bigben/32/7981_2.png) [@bigben](https://discuss.elastic.co/u/bigben)\
**Post date:** [November 26, 2019, 3:28am UTC](https://discuss.elastic.co/t/logstash-lost-data-when-using-file-input-plugin/209406/1 "2019-11-26T03:28:46Z")

</div>

Hi all,

I use the logstash file input plugin, and data lost occur.

Every 1 hour, download log files to the path ("/path/to/.../_.log._"), and I use the logstash file input plugin to tail them.

When Logstash started, it could read files normally, but few hours later, logstash missing read data, like this graph:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/c/bcbe60374451eb411dea0377a23e36888a203068.png)  
I don't know what situation cause this, it's normally on my old elastic stack on Windows server (1 es node), like following,

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68b8fca4e1f42cb57db38c37d4f808119bd12682.png)  
but the new elastic stack on the Centos server (3 es nodes), become abnormal.

The difference between old and new elastic stack environment:

1. OS, the old one is Windows, and new one is Centos.

2. Elastic stack version, old: 6.7.2; new: 7.4.2

3. The Elasticsearch node nimber, old: single node; new: 3 nodes

4. In the old environment, download log files to the path on Windows server by day (30 1 \* \* \*); in new environment on Centos server, by every hours (30 \* \* \* \*).

5. Used the "index lifecycle management" to house keeping the log data on new environment.

Here is my Logstash conf:

```
input {
  file { 
    type => 'log'
    mode => "tail"
    path => "/path/to/.../*.log.*" 
    codec => multiline {
      pattern => "^\d\d\d\d\-\d\d"
      negate => true
      what => "previous"
      auto_flush_interval => 1
    }
    start_position => "beginning"
    ignore_older => "12 h"
    sincedb_clean_after => 1
  }
}

filter {
  if [type] == 'log'
  {
    if [message] == "" { drop{} }
    mutate {
      ...
    }
    csv {
      ...
    }
    grok{
     ...
    }
    kv {
      ...
    }
    date {
      ...
    }
    ruby {
     ...
    }
    jdbc_static {
    ...
    }
    if "_dateparsefailure" in [tags] { drop{ } }
}

output{
  if [type] == 'log'
  {
    elasticsearch {
      hosts => ["es1", "es2", "es3"]
      index => "log-%{+YYYY.MM.dd}"
      ilm_rollover_alias => "log"
      ilm_policy => "log-house-keeping"
    }
  }
}

```

The sindb file generated by logstash file input plugin:

 ![%E6%93%B7%E5%8F%96](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e28b892a20149cea7248cd7087ce4f6e88d5e698.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 24, 2019, 3:28am UTC](https://discuss.elastic.co/t/logstash-lost-data-when-using-file-input-plugin/209406/2 "2019-12-24T03:28:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.