Transform error handling

[liorg2]

hi there,
is there a way to configure a transform to ignore errors?
it seems that if it fails, it will changed the state the failed state, and stop the processing.

example:
imagine I define a transform for data-.
data- index contains 2 fields: 1. server- keyword, 2. cpu- float.
the transform just run a grou by server, and avg of cpu.
when it ran over index data-xx, it found that the server field was defined with a text data type instead of keyword, and the terms aggregation failed.

how can I ignore this error?

[Hendrik_Muhs]

Transform differs between 2 sorts of errors:

• temporary ones, e.g. timeouts, network outages to the CCS remote, etc.
• permanent errors, which make it impossible to continue, e.g. script errors, broken mappings, etc.

Yours is a permanent one, the search will always fail, there is no reason to continue. You have to fix your config/mappings.

[liorg2]

I understand but in my case it doesn't matter as it's one index out of many.. I would consider to allow configuration of error handling policy

[Hendrik_Muhs]

A design goal of transform was to ensure data consistency and correctness. I think your proposal is incompatible with that. Because we do not retrieve data 1 by 1 but in batches, it would mean we loose a whole page, so we would also loose other valid data point at random.

For solving mapping problems like yours, its better to fix ingest and reindex the broken indexes.

Another solution for your special case: You could use a script to workaround the problem in your terms aggregation. However scripts might not be compatible in continuous mode.