Stopping trying to execute pipeline for document not on current index

Elastic Stack Elasticsearch
1

I've been looking into one of the most recent log file (.i.e the log entries for one day) and I've found about 295 log entry similar to the following:

[DEBUG][o.e.a.b.TransportBulkAction] [es-1] failed to execute pipeline [lgw] for document [index-2019.06.27/doc/null]

There are even entries for document in index which are already long gone!

How can I stop pipeline execution, if there are not at most directed to the index of the day before the current day?

2

hey,

how about a processor that checks in your pipeline for the date of the timestamp that the document provided? If that is older than X, you could use the drop processor.

--Alex

3

That could be interesting, is there a way to set that globally? I have about half a dozen pipeline I'll need to update.

4

no, pipelines can be configured on a per-index base though.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.