Merge two index to create third index using logstash

What I have done so far
input {
elasticsearch {
hosts => "localost"
index => "employees_data,transaction_data"

     query => '{ "query": { "match": { "code": 1} } }'
    scroll => "5m"
    docinfo => true
  }
}
output {

elasticsearch {
hosts => ["localhost"]

index => "join1"
   }

}

It's giving me output like this
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [
{
"_index" : "join1",
"_type" : "doc",
"_id" : "72gIv3QB_L6Y9V8lNpCh",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"@timestamp" : "2020-09-24T07:33:40.421Z",
"payment" : 32080,
"moth" : "june",
"code" : 1
}
},
{
"_index" : "join1",
"_type" : "doc",
"_id" : "8GgIv3QB_L6Y9V8lN5AG",
"_score" : 1.0,
"_source" : {
"city" : "indore",
"@version" : "1",
"@timestamp" : "2020-09-24T07:33:40.408Z",
"name" : "Abhi",
"salary" : 320800,
"code" : 1
}
}
]
}
}

How to get it in third index but one record based on code field

I think logstash might not be the right tool, it's conceptually a mapper, what you need is a reducer. In practice you want to group documents, in your case you want to group docs with the same code field. You need aggregations for that.

Long story short, please have a look at the very similar ask: Merging documents based on matched fields values

As you explicitly say that you want an index as output, transform sounds like the right tool to me.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.