How to convert logstash elasticsearch plugin in ingest pipeline?

[crpj]

Hello,

I would like to know if and how it is possible to use the logstash elasticsearch plugin in an ingest pipeline, currently the use of a workaround in logstash which enables enriching data based in a query from another index works fine as follows:

#logstash excerpt
filter {

    elasticsearch {
	  hosts => "http://localhost:9200/"
      query_template => "C:/Elastic/get_key_query.json"
      index => "ref-data-keys"
      fields => {
        "KEY" => "KEY"
      }
      remove_field => ["host", "@version", "@timestamp"]
    }
}
}
output {
	elasticsearch {
		hosts => "http://localhost:9200/"
		index => "destination-index-%{+YYYYMM}"
  }
}

#json query found in a separate file
{
  "size": 1,
  "_source": ["KEY"],
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "PRODUCT.keyword": "%{PRODUCT}"
          }
        },
        {
          "match": {
            "COMPONENT.keyword": "%{COMPONENT}"
          }
        }
      ]
    }
  }
}

It is possible to replicate this logic in ingest pipeline?

Thank you.

[stephenb (Stephen Brown)]

Hi @crpj

Yes there is there is even a special processor to enrich data in it ingest. See Here

Enrich your data | Elasticsearch Guide [7.13] | Elastic

[crpj]

Thank you @stephenb

I have seen the Enrich feature once, if I understood it right, Enrich can only work with a single field for the lookup. Then I had the workaround I mentioned, the way I did with logstash and the elasticsearch plugin allows me to create a more complex query based in several fields (e.g. PRODUCT, COMPONENT and many more) to find a value (e.g. Key) that will be used to enrich the destination field.

My question is whether it is possible to replicate the same workaround I posted above directly in an ingest pipeline, I believe it is not possible because it references a json query file but wanted to confirm. The objective is to use Transforms and associate an ingest pipeline with that more complex enrich logic.

ps: that workaround was based in @dadoonet solution (thanks by the way)
https://david.pilato.fr/blog/2018/03/22/enriching-your-postal-addresses-with-elastic-stack-part-1/

[stephenb (Stephen Brown)]

Well you could concatenate the the PRODUCT and COMPONENT into a single term field at ingest time, and use that as a lookup in the enrich, you would need to do that in the lookup data as well, I did that for a very similar case worked great... if you need to use a query then no, enrich processor is probably not the correct approach.