jsheely
July 1, 2024, 6:18pm
1
I have been seeing several commits, PR, and change log release notes, discussions around failure_store .
But I can't find anything in the documentation about it and when I try and utilize it based on the examples I am getting failures about unknown parameter.
Has this feature been released? If is there an example of how to utilize it?
I am currently using version 8.14.1
References:
elastic:main ā jbaiera:data-stream-ingest-failure-redirect
opened 09:43PM - 14 Dec 23 UTC
This PR updates the ingest service to detect if a failed ingest document was bouā¦ nd for a data stream configured with a failure store, and in that event, restores the document to its original state, transforms it with its failure information, and redirects it to the failure store for the data stream it was originally targeting.
Example run with a default pipeline and data stream:
```json
PUT _ingest/pipeline/testpipeline
{
"processors": [
{
"fail": {
"message": "This test pipeline fails for all documents"
}
}
]
}
PUT _index_template/my_data_stream_template
{
"index_patterns" : ["my_data_stream*"],
"data_stream": {
"failure_store": true
},
"priority" : 1,
"template": {
"settings" : {
"number_of_shards" : 1,
"index.default_pipeline": "testpipeline"
}
}
}
POST my_data_stream_1/_doc
{
"key": "value",
"@timestamp": "2023-12-14T12:00:00Z"
}
>>>
{
"_index": ".fs-my_data_stream_1-2023.12.14-000001",
"_id": "8es0aowBHIk1gE8HEwcs",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}
POST .fs-my_data_stream_1-2023.12.14-000001/_search
>>>
{
"took": 47,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": ".fs-my_data_stream_1-2023.12.14-000001",
"_id": "8es0aowBHIk1gE8HEwcs",
"_score": 1,
"_source": {
"@timestamp": "2023-12-14T21:20:46.566Z",
"document": {
"index": "my_data_stream_1",
"source": {
"@timestamp": "2023-12-14T12:00:00Z",
"key": "value"
}
},
"error": {
"type": "fail_processor_exception",
"message": "This test pipeline fails for all documents",
"stack_trace": "org.elasticsearch.ingest.common.FailProcessorException: This test pipeline fails for all documents\n\tat org.elasticsearch.ingest.common@8.12.0-SNAPSHOT/org.elasticsearch.ingest.common.FailProcessor.execute(FailProcessor.java:41)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:165)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:141)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:129)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.ingest.IngestDocument.executePipeline(IngestDocument.java:867)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.ingest.IngestService.executePipeline(IngestService.java:1020)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.ingest.IngestService.executePipelines(IngestService.java:879)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.ingest.IngestService$1.doRun(IngestService.java:765)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.12.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"
}
}
}
]
}
}
```
I would say that this kind of mapping conflict errors are pretty common, if you search on the forum you will find a lot of topics about this, but in my opinion the root cause for this is a misconception on how Elasticsearch works.
Elastiscearch is not schema-less, it is schema-on-write and has some features to also have schema-on-read (like runtime fields), but having a schema (the mappings) it is a requirement.
There are some ways to avoid the issues of conflicting mappings, but in the end tā¦
TimV
July 4, 2024, 2:49am
2
No, it has not been released, which is why you can't find any documentation for it.
2 Likes