Create an error log with an ingest pipeline when there is an indexing error

Elastic Stack Elasticsearch
1

Hi,
I would like to create an error log when a document is not suitable for my mapping.

For example if I send a document containing an unknown field in my mapping (dynamic:false), I would like to be able to create an error log in another index.

Is it possible with an ingest pipeline ?

2

Ingest pipelines | Elasticsearch Guide [7.15] | Elastic touches on that, specifically the on_failure parameter.

3

Okay, thx for the doc.
But the on_failure parameter work with a processor, which one can i use ?
I just want to test the incoming documents, et create an error when a field is unknow.

4

Unless there is something not well explained in the documentation, I don't think it is possible.

The on_failure parameter applies to errors on processors in the ingest pipeline, it allows you to deal with errors during the ingest pipeline processing, the mapping error will happen after that phase, when Elasticsearch tries to index it.

5

Okay,

So there is no way to know if a document has an unknown field before indexing it ?
With an ingest pipeline if possible.

6

Maybe you can use the script processor to write a painless script that will test each field and see if they are in a list of allowed fields, but I do not have experience with painless script to know how you can do that.

But I don't think this is something easy to do, and also it can impact the performance.

7

Thanks, this is a good idea.

I searched how to get the field names of the documents, in painless, but no result.
If you know how to do this...

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.