Enrich documents by copying fields from another index

Elastic Stack Elasticsearch
8

Guys,

Actually, the feature you're asking for is coming in 7.5 via the new enrich processor, which kind of provides index-time JOIN capability.

The main idea is to set up an enrich policy that will source data from your related indexes into a new "enrich index" and then you can leverage that "enrich index" in your ingest pipeline using an enrich processor in order to enrich your documents with related fields.

So, without going in too many details, here is how it works in practice:

  1. You have an index A with fields (a, b, c, d) that you'd like to use for enriching your incoming documents
  2. You define an enrich policy based on that index A and the "join" field a
  3. You define an ingest pipeline with an enrich processor that will try to match field z of the incoming document against field A.a of the enrich index
  4. If a match is found, your incoming document will get fields b, c and d from the index A. Note that it will also get the match field a that you can remove using a remove processor if needed.

That should pretty much work the way you expect. You can find a complete example here. At the beginning, it will work for exact matches (i.e. term query) and geo matches (i.e. geo_shape query), but they will probably add new kind of matches (like range matches) in the near future.

1 Like