How to remove duplicate values?

Elastic Stack Logstash
1

I've got serveral docs with same "index" and many duplicate values in elasticsearch:

{
"_index" : "bahnlog-29_11_2019",
"_type" : "_doc",
"_id" : "l7YfrG4Baw1619OK5ufn",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2019-11-27T09:10:00.816Z",
"@version" : "1",
"parsed" : {
"Journey" : [
{
"ID" : "2019-11-27T09:59:00.000Z_8000191",
"delay" : 14
},
{
"ID" : "2019-11-27T09:59:00.000Z_8000231",
"delay" : 2
}
]
{
"_index" : "bahnlog-29_11_2019",
"_type" : "_doc",
"_id" : "mLZErG4Baw1619OKhufy",
"_score" : 1.0,
"_source" : {
"parsed" : {
"Journey" : [
{
"ID" : "2019-11-27T09:59:00.000Z_8000191",
"delay" : 14
}
{
"ID" : "2019-11-27T09:59:00.000Z_2323231",
"delay" : 1212
}

How can I remove the whole duplicate fields which belogs to the same ID or maybe merge the two docs? To always overwrite the doc with a document_id is not the solution in my opinion, because I want to log the old values as well.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.