Trying to pass mongodb id as elasticsearch id

Elastic Stack Logstash
1

Logstash 7.8
We have a case where we collect logs from a mobile app using parse-sdk.
The data is sent to a MongoDB replicaset.
Logstash is configured using the mongoschema jdbc driver to pull the data from Mongo and push it to elasticsearch.

Currently however we have the same event being logged twice on occasion even after setting fingerprint with UUID.

In our Mongo document we can see a unique _id field which becomes document._id in elasticsearch.

We tried to set the mongo _id to the elasticsearch _id but it all we can see in the _id field(elasticsearch) is the variable that we set %{[_id]}

Also tried a mutate filter to first rename from _id to mongo_id to no avail.

Code below -

filter {

mutate {
  rename => [ "_id", "mongo_id" ] #Also tried to change the _id field to document._id
}

#fingerprint {
#  source => "[document._id]"
#  target => "[@metadata][fingerprint]"
#  method => "MD5"
#  key => "1234"
#}

}

output {
elasticsearch {
    hosts => ["http://127.0.0.1:9200"]
  #  document_id => "%{[@metadata][uuid]}"
  #  document_id => "%{[@metadata][fingerprint]}"
  #  doc_as_upsert => true
    document_id => "%{[mongo_id]}"
    codec => json
   }
}

Please help

2

no need to rename just simple add array as document [document][_id]

output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
doc_as_upsert => true
document_id => "%{[document][_id]}"
codec => json
}
}

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.