Logstash not working with allow_time_override and getting data loss influxdb

Elastic Stack Logstash
1

I am facing data loss issue with logstash version 5.2 with influxdb output pluggin.

My logstash configs are:

        filter {
          grok {
         patterns_dir => ["/etc/logstash/patterns"]
              match => { "message" => "%{NGINXACCESSBLOG}"}
              if [type] == "nginx-access-router" {
            ruby { code => "event.set('epoc', event.get('@timestamp').to_i)" }
            date {
                match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
            target => "newdate"
              }

            mutate { convert => [ "request_time", "float" ] }
            if ![response] {
              mutate { add_field => ["response", "nil"] }
            }
        }
        }
        }


    output {

      if [type] == "nginx-access" {

            influxdb {
          host => "localhost"
          port => 8086
          user => "admin"
          password => "XXX"
          db => "xx"
          allow_time_override => true
          retention_policy => "XX"
          measurement => "XXXX"
          enable_metric => false
          send_as_tags => ["response"]
          data_points => {

            "response" => "%{[response]}"

        "timestamp" => "%{[epoc]}"
            "time" => "%{[epoc]}"


         }
         coerce_values => {
            "request_time"   => "float"
        }
    }
    }
    }

Note: I have checked in debug log that all received log lines are successfully parsed with grok pattern.

Then why getting more then 50% data loss.

Some solution i tried: As per https://github.com/logstash-plugins/logstash-output-influxdb/issues/69

I tried to use "allow_time_override" as above mentioned configuration but logstash stop pushing data to influxdb and getting nothing in error log file.

2

After lots of tracing step by step able to find the root cause and fix of data loss issue as i posted yesterday.

Issue root cause is:

Influxdb overrides duplicate timestamp
To know more Why influxdb does override duplicate timestamp entry

Add unique filter as:

uuid {
target => "uuid"
overwrite => true
}

Then add in output tag as:

send_as_tags => ["uuid"]
      data_points => {
        "uuid"=>"%{[uuid]}"
       }
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.