Merge or combine two indexes to one based on common field (id)

Elastic Stack Logstash
1

I have one index (index1) with columns "name" and "id" and the other index (index2) with columns "resume" and "id".
"id" is the common field between two indexes.
How can I merge or combine them together so I get one index (index3) with columns "name", "resume", "id".
When I say combine, it means if I have data in index1 as
"name": "Mike"
"id": 123
and I have data in index 2 as
"resume": "looks for ...bla bla bla.."
"id": 123

This is what I want in index 3 as a result of combining index1 and index2:
index3:
"name": "Mike"
"id": 123
"resume": "looks for ...bla bla bla.."

2

I think you can use input filter named elasticsearch
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

use two input filter,
and then output on elasticsearch with new index name

3

I am using the following configuration but I am getting "connection refused" error:
input {
elasticsearch {
hosts => "localhost"
user => "abc"
password => "abc1"
index => "twitter1"
size => 500
scroll => "5m"
docinfo => true
}
}
output {
elasticsearch {
document_id => "%{[@metadata][_id]}"
hosts => "localhost"
ssl_certificate_verification => "true"
index => "twitter3"
user => "abc"
password => "abc1"
}
}
I think the issue is "hosts" in elasticsearch in input, but I don't know what to put there.

4

Hi did you fixed this issue
I think you should have
input {
elasticsearch {
hosts => ["hostname:9200"]
index => "index_name"
}
}

5

I tried. It didn't work. I am getting error:
"Failed to open TCP connection to hostname:9200 (initialize: name or service not known)"

6

is your elasticsearch reading via name or localhost.
what is output of following from your system

curl hostname:9200/_cluster/health?pretty

7

The problem was certificate. Now it is working. Thank you for the follow up.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.