General questions about ingest

[pdesoyres (Pierre De Soyres)]

I have general questions about the ingest feature in elasticsearch.

My need is to add several informations to the documents I index. Those informations are taken from external services reachable through REST APIs. I need to call many services, each of them will add one or more fields to my document. Each services can take a long time to execute.

I wanted to know if ingest feature is designed for performing such a long process.
Does the ingest pipeline execution is blocking or is it done in background ?
Can I write a custom ingest processor plugin that will perform the call to a REST API ?

Pierre

[dadoonet (David Pilato)]

Not really designed for that but I started a similar experience some months ago by running searches from an ingest processor.
This went pretty well but as you said if the service takes a looooong time to answer it will somehow block ingest nodes.

You can see a similar effort here: https://github.com/elastic/elasticsearch/pull/20340

I'd be super careful in writing such a component like Simon said at https://github.com/elastic/elasticsearch/pull/20340#issuecomment-245064831

[pdesoyres (Pierre De Soyres)]

Thank you David,

this is very clear. I understand why it is not a good option.