Merging document _source from 2 indices to a 3rd index

Elastic Stack Elasticsearch
1

I have a couple of indices with the following document structure.

Index 1
"_source": {
"guid": "abc123",
"documentdescription": "document description",
}

Index 2
"_source": {
"guid": "abc123",
"documenttitle": "document title",
}

I want to create a 3rd index by with _source with merged attributes on guid.

Index 3
"_source": {
"guid": "abc123",
"documentdescription": "document description",
"documenttitle": "document title",
}

I have explored logstash ingestion to create a new index (Index 3) however was not able to figure out a technique to accomplish this. Trying to avoid writing a full blown client program to loop over all documents in both indices :slight_smile:

2

With logstash, I believe you can use elasticsearch-input plugin to read all the docs from index 1 then elasticsearch-filter plugin to do lookups in index 2 and elasticsearch-output plugin to write to index 3.

3

Thanks David, Some sample code will be a huge :slight_smile:

6

I'm sorry it took so long but here is in short what you can do:

input {
  elasticsearch {
    index => "index1"
  }
}

filter {
  elasticsearch {
    index => "index2"
    query => "guid:%{[guid]}"
    fields => {
      "documenttitle" => "[documenttitle]"
    }
  }
}

output {
  elasticsearch {
    index => "index3"
  }
}

Something along those lines. I did not test it though.

Hope this helps.