Call to external application using scrip processor in ingest pipeline

Elastic Search
1

Hi there. It is possible to call an external application/WebService/REST API from a script processor as part of an ingest pipeline?
We would like to do some custom processing and considering the option of calling an external endpoint as part of the ingest pipeline. The script could be in painless if possible.
Thanks in advance

2

Saw some resplies one some similar inqueries, but some years old, so I am curious if this still applies.
Reasoning about not slowing or even blocking the ingest process makes perfect sense.
Little more background.
Intend is to use the SharePoint connector to get the documents. Some of the documents must be transcribed using a multimodal visual LLM into text before ingesting to ES.

Something like this, where the custom module is then sending the image data to a Visual LLM:

image
image960×540 41.9 KB

3

This is a use case for Logstash where you can use the http filter to integrate with external services during the ingestion process.

With ingest pipelines it is not possible.

1 Like
4

Thanks for clarifying this. I will check out the Logstash option.

5

An alternative here, depending on your exact needs, would be to load data from the external module into an index in Elasticsearch and then use the enrich processor to load data from the external module into the document and then making filtering/processing/dropping decisions based on those enriched values that are now present in the document.

6

The source documents are stored in a SharePoint repository and we have been aiming at using the Elastic Search connector for SharePoint to fetch the documents. Some of the documents are Powerpoint slide decks with many graphical elements and these has to be translated into text before being added to the vector index and made searchable.
So what you are suggesting is to load all the documents with graphical elements, such as PowerPoints into a separate index and from there call the visual model before merging into the final document represantation in a separate index?

7

Your use case sounds a bit more complex than I was thinking so my suggestion probably isn't relevant here