# Logstash not working with allow\_time\_override and getting data loss influxdb

**URL:** <https://discuss.elastic.co/t/logstash-not-working-with-allow-time-override-and-getting-data-loss-influxdb/132488>\
**Category:** Logstash\
**Created:** [May 18, 2018, 12:27pm UTC](https://discuss.elastic.co/t/logstash-not-working-with-allow-time-override-and-getting-data-loss-influxdb/132488 "2018-05-18T12:27:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Muktesh\_Kumar](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@Muktesh\_Kumar](https://discuss.elastic.co/u/Muktesh_Kumar)\
**Post date:** [May 18, 2018, 12:27pm UTC](https://discuss.elastic.co/t/logstash-not-working-with-allow-time-override-and-getting-data-loss-influxdb/132488/1 "2018-05-18T12:27:09Z")

</div>

I am facing data loss issue with logstash version 5.2 with influxdb output pluggin.

My logstash configs are:

```
        filter {
          grok {
         patterns_dir => ["/etc/logstash/patterns"]
              match => { "message" => "%{NGINXACCESSBLOG}"}
              if [type] == "nginx-access-router" {
            ruby { code => "event.set('epoc', event.get('@timestamp').to_i)" }
            date {
                match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
            target => "newdate"
              }

            mutate { convert => ["request_time", "float"] }
            if ![response] {
              mutate { add_field => ["response", "nil"] }
            }
        }
        }
        }

    output {

      if [type] == "nginx-access" {

            influxdb {
          host => "localhost"
          port => 8086
          user => "admin"
          password => "XXX"
          db => "xx"
          allow_time_override => true
          retention_policy => "XX"
          measurement => "XXXX"
          enable_metric => false
          send_as_tags => ["response"]
          data_points => {

            "response" => "%{[response]}"

        "timestamp" => "%{[epoc]}"
            "time" => "%{[epoc]}"

         }
         coerce_values => {
            "request_time" => "float"
        }
    }
    }
    }

```

Note: I have checked in debug log that all received log lines are successfully parsed with grok pattern.

Then why getting more then 50% data loss.

Some solution i tried: As per [https://github.com/logstash-plugins/logstash-output-influxdb/issues/69](https://github.com/logstash-plugins/logstash-output-influxdb/issues/69)

I tried to use "allow\_time\_override" as above mentioned configuration but logstash stop pushing data to influxdb and getting nothing in error log file.

---

<div class="post-metadata">

**Author:** ![Muktesh\_Kumar](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@Muktesh\_Kumar](https://discuss.elastic.co/u/Muktesh_Kumar)\
**Post date:** [May 20, 2018, 11:29am UTC](https://discuss.elastic.co/t/logstash-not-working-with-allow-time-override-and-getting-data-loss-influxdb/132488/2 "2018-05-20T11:29:12Z")

</div>

After lots of tracing step by step able to find the root cause and fix of data loss issue as i posted yesterday.

Issue root cause is:

Influxdb overrides duplicate timestamp  
To know more Why influxdb does override duplicate timestamp entry

> **[Why influxdb does override duplicate timestamp entry | Techathon](http://techathon.techinnolab.com/why-influxdb-does-override-duplicate-timestamp-entry/)**
>
> In InfluxDB, An entry is uniquely identified by the measurement name, tag set, and timestamp. If you are inserting a new entry with the same measurement, tag set, and timestamp as an existing point. ...

Add unique filter as:

```
uuid {
target => "uuid"
overwrite => true
}

```

Then add in output tag as:

```
send_as_tags => ["uuid"]
      data_points => {
        "uuid"=>"%{[uuid]}"
       }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 17, 2018, 11:29am UTC](https://discuss.elastic.co/t/logstash-not-working-with-allow-time-override-and-getting-data-loss-influxdb/132488/3 "2018-06-17T11:29:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.