I would like to know if and how it is possible to use the logstash elasticsearch plugin in an ingest pipeline, currently the use of a workaround in logstash which enables enriching data based in a query from another index works fine as follows:
I have seen the Enrich feature once, if I understood it right, Enrich can only work with a single field for the lookup. Then I had the workaround I mentioned, the way I did with logstash and the elasticsearch plugin allows me to create a more complex query based in several fields (e.g. PRODUCT, COMPONENT and many more) to find a value (e.g. Key) that will be used to enrich the destination field.
My question is whether it is possible to replicate the same workaround I posted above directly in an ingest pipeline, I believe it is not possible because it references a json query file but wanted to confirm. The objective is to use Transforms and associate an ingest pipeline with that more complex enrich logic.
Well you could concatenate the the PRODUCT and COMPONENT into a single term field at ingest time, and use that as a lookup in the enrich, you would need to do that in the lookup data as well, I did that for a very similar case worked great... if you need to use a query then no, enrich processor is probably not the correct approach.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.