Duplicate Lines

Elastic Stack Logstash
1

Hi all,

I have a problem with my ELK configuration.
I use :

  • logstash 2.4.X
  • Elasticsearch : 2.4.1
  • Kibana : 4.6.1

My Logstash.conf

input {
  file {
    type => "MyAppLogs"
    path => "/var/log/myApp/*.log"
  }
}

filter {

  uuid {
    target => "@uuid"
    overwrite => true
  }
  fingerprint {
    source => ["message"]
    target => "fingerprint"
    key => "78787878"
    method => "SHA1"
    concatenate_sources => true
  }
 (some Grok patterns)
}

output {
if "_grokparsefailure" not in [tags] {
  elasticsearch {
    hosts => ["elasticsearch.host.ip"]
    document_id => "%{fingerprint}"
    }
  }
}

If i check in my elasticsearch instance, i see three lines for juste one log line :

CaptureElasticsearch.JPG936×57 27.8 KB

As you can see, i generate a fingerprint and try to replace the document_id by my fingerprint.
If i look inside these three lines, i see that they have the same fingerprint (94c475dfd211a587c34610eaa96df1b823a26f3d) but only one line have his document_id replace by the fingerprint.

Do you have any idea why i have these duplpicate lines ?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.