Enrich documents by copying fields from another index

Hi,

I got 2 indices.
say index1 and index2.

When I index a new document in index2, I want some fields to be copied from the document in index1 having the same _id.

I previously realized this using logstash + the elasticsearch filter.

But this time, i'm not using logstash. I index documents in index2 using the bulk api.

What is the best way to achieve that ? Ingestion plugin ?

Thanks.

What is the best way to achieve that ?

I think that what you did in the past is a good way to solve this problem. I mean:

I previously realized this using logstash + the elasticsearch filter.

That's probably what I'd do.

Thanks @dadoonet but I want to get rid of logstash exactly as you describe here :

I would like to include in the ingest node pipeline some plugin that is able to enrich datas from existing documents in the cluster.

Yeah. I'm not doing that talk anymore. That ended up being a bad idea after all to use ingest to do lookups in a 3rd party system. I was more using that for demo purpose, to introduce what ingest is and how you can easily build your own plugin but the use case was not the best one. :frowning:

You can see what is my recommendation today (using Logstash):

If you really want to go that way (which I do not recommend), you can look at this PR which was a WIP but was never merged because of the reasons I exposed before.

@dadoonet Thanks for the share.

I will also reconsider using logstash.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.