As part of my project's workflow, we have a pipeline which uses the Elasticsearch filter plugin a few times in order to find a set of items which need to be deleted from several indices. This always works perfectly when called manually for a single data input, but when called as part of a large data submission, we end up with a 'Read timed out' error ('Failed to query Elasticsearch for previous event') on some of those Elasticsearch filter queries.
From this thread, I learned that the Elasticsearch filter is rather heavy, which explains why we occasionally get this error when the pipeline is called many times for a large data ingestion, but unfortunately no solution was proposed:
Elasticsearch input missing [@metadata][_index] - Elastic Stack / Logstash - Discuss the Elastic Stack
While the Elasticsearch input and output plugins seem to have various timeout options, I cannot see one for the Elasticsearch filter plugin. We've not set a global timeout for queries in our Elasticsearch instance, so is there any way to resolve this without completely re-organising how we do things?