Hi,
I have applied the suggested technique to prevent from duplicates creation in the Logstash output plugin for Elasticsearch, as described here: https://www.elastic.co/blog/logstash-lessons-handling-duplicates
Basically I associate to the _id field, the id field provided by the application generating the messages (id is the timestamp at nanotime):
output {
elasticsearch {
hosts => "host_IP"
document_id => "%{[id]}"
}
}
Therefore I would expect to see in Kibana the same value for _id and id.
This correct association happens in the most of the cases but sometimes I notice that, in the Kibana Discover view, the _id field differs by 1 respect to the id field. Here is an example (in the Kibana index pattern _id is type string and id is type number):
_id= 11335773562862155 (correct value)
id= 11,335,773,562,862,156 (wrong value)
Please note that I write correct or wrong above with respect to the original message being published to Kafka, before hitting the ELK stack.
Moreover if I search for this specific document filtering by id, I obtain a result only if I search for
id = 11335773562862155. This induce me to think that the association from the Logstash plugin works as expected but somehow the value is not properly displayed by the Kibana interface.
Does anyone have an idea about the origin of this incorrect display in the Kibana Discover view? Is it some Javascript casting issue?
