Ingest Pipeline - Why Isn’t My Failure Processor Redirecting Failed Docs to a New Index?

Elastic Stack Kibana
1

Following the Elastic documentation for Handling pipeline failures I have added a failure processor which is supposed to change the _index, but it does not seem to work as I thought.

Failure Processors (.json) 
[
  {
    "set": {
      "field": "_index",
      "value": "failed-{{{ _index }}}",
      "description": "Index document to 'failed-<index>'"
    }
  },
  {
    "set": {
      "field": "error_information",
      "value": "Processor {{ _ingest.on_failure_processor_type }} with tag {{ _ingest.on_failure_processor_tag }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}",
      "ignore_failure": true,
      "description": "Record error information"
    }
  }
]

image
image1166×197 12.8 KB

My Pipeline stats show that this pipeline's grok fails often:

image
image1709×172 13.3 KB

I was expecting to find an index named: failed_<something> with the failing documents, but I cannot find anything.

Anyone able to spot the issue, what am I misunderstanding here?

1 Like
2

Is this from some Elastic Agent integration or a custom ingestion?

1 Like
3

Our Elastic Agent uses the Custom Logs (Filestream) integration where we have configured it to use our self-made Ingest Pipeline we have configured/created in Kibana.

image

1 Like
4

Yeah, this is a permission issue, Elastic Agent policies have pretty limited permissions, they can only write to the data streams for their integration, so using set or reroute processors to change the name of the index will not work because the API Key used by the policy will not have the permissions required.

In your example is more complicated because Elastic Agent managed by fleet cannot write into anything that does not follows the Elastic data stream naming scheme, so for logs you can only write into datastreams starting with logs-*, trying to write into failed_* will not be possible.

What is the version of your stack? If you are at least on 9.1 you can now add extra permissions to extra datastreams in the policy, but you would need to use logs-* something for this.

Also, on 9.1+ you have the option to automatically capture failed events with the Failure Store.

2 Likes