Merge two index to create third index using logstash

Elastic Stack Logstash
1

What I have done so far
input {
elasticsearch {
hosts => "localost"
index => "employees_data,transaction_data"

     query => '{ "query": { "match": { "code": 1} } }'
    scroll => "5m"
    docinfo => true
  }
}
output {

elasticsearch {
hosts => ["localhost"]

index => "join1"
   }

}

It's giving me output like this
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [
{
"_index" : "join1",
"_type" : "doc",
"_id" : "72gIv3QB_L6Y9V8lNpCh",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"@timestamp" : "2020-09-24T07:33:40.421Z",
"payment" : 32080,
"moth" : "june",
"code" : 1
}
},
{
"_index" : "join1",
"_type" : "doc",
"_id" : "8GgIv3QB_L6Y9V8lN5AG",
"_score" : 1.0,
"_source" : {
"city" : "indore",
"@version" : "1",
"@timestamp" : "2020-09-24T07:33:40.408Z",
"name" : "Abhi",
"salary" : 320800,
"code" : 1
}
}
]
}
}

How to get it in third index but one record based on code field

2

I think logstash might not be the right tool, it's conceptually a mapper, what you need is a reducer. In practice you want to group documents, in your case you want to group docs with the same code field. You need aggregations for that.

Long story short, please have a look at the very similar ask: Merging documents based on matched fields values

As you explicitly say that you want an index as output, transform sounds like the right tool to me.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.