Enriching data in index by data from another index

[jneuschl (JN)]

Hi guys,
we have a need to enrich user data (e.g uid=19212012) coming from all applications to include additional data about user (e.g branch=xxx;region=yyy).

At first we wanted to use static jdbc in logstash, but security prohibited us to use that (complex reason to explain why ).

Wy have data needed for such enrichment in elasticsearch index.

Is there a way how to preprocess message to include data based on UID from another index?

Another option that would be acceptable is to enrich data during rollup jobs (if that's possible).

Many thanks

[spinscale (Alexander Reelsen)]

Hey,

another possible option might be the ingest node, which is able to change the JSON format before the data gets indexed. However you cannot merge data from another index, but maybe you can use a lookup table in a script processor for example.

--Alex

[jneuschl (JN)]

Hi there,

that could be a solution however can you plz advise how to solve following:

• in examples for elasticsearch filter, is just a simple setup without authorization, does it work with basic auth as well?
• can data from elasticsearch be cached? We have about 300 incoming record a second
• any other possible solution ?

Thanks