Ingest Pipeline Failure Processor Shard Failures

[paolovalladolid]

I set up an ingest pipeline in Dev Tools console with this command

PUT _ingest/pipeline/ams-log-pipeline
{
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{@timestamp} %{logLevel} %{className} %{messageContent} %{}",
        "if": "ctx?.fields?.app_id != null && ctx.fields.app_id != 'ams-kafka-consumer'",
        "ignore_failure": true
      }
    },
    {
      "dissect": {
        "field": "message",
        "pattern": "%{logDate} %{logTime} %{logLevel} %{className} %{task} CACHE_MANAGER_RESPONSE %{kafkaTopic} %{statusCode} %{statusString}",
        "if": "ctx?.fields?.app_id != null && ctx.fields.app_id == 'ams-kafka-consumer'",
        "ignore_failure": true
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "description": "Record error information",
        "field": "error_information",
        "value": "Processor {{ _ingest.on_failure_processor_type }} with tag {{ _ingest.on_failure_processor_tag }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}"
      }
    }
  ]
}

The ingest pipeline appears to let through documents that its 2 processors cannot deal with instead of just failing and letting nothing through - this is good for us.

However, I see shard failures when I run queries in Kibana on documents that were ingested through this pipeline. The failure processor is not populating the field error_information, which I understand should have details on what went wrong with the 2 Dissect processors.

I know what an illegal argument exception is but in this case I just can't figure out what is throwing it

What do I need to do to fix this and eliminate the shard failures?

[jsanz (Jorge Sanz)]

Have you tried the Simulate API with a failing sample document to check the appearance of the resulting document?

Simulate pipeline API | Elasticsearch Guide [8.7] | Elastic

Where are those shards failures happening? Is it on Discover or a Visualization?

[paolovalladolid]

Thanks Jorge!

The failures were on a dashboard that included visualizations.

I clicked the Request and Response tabs and found the illegal_argument_exception was caused by one of the text fields in the document.

The document had been transformed by our ingest pipeline. The text field was one of the fields generated by a Dissect processor in the pipeline. The same field was being used by a Top Values function in one of the visualizations The error message recommended replacing
"fieldname" with "fieldname.keyword" as one of the fixes. I did that and the shard failures disappeared.

[paolovalladolid]

One last question (I hope) on this thread.

We are using one Filebeat instance to ship logs from six different log files.

This instance is shipping the logs to one ingest pipeline, with 4 Dissect processors to extract fields and add them to applicable documents.

From what I can tell, only one pipeline at a time can be configured in the output.elasticsearch section of filebeat.yml. Is this correct?

[tsullivan (Tim Sullivan)]

Hi @paolovalladolid

If you use the pipelines option in the elasticsearch.output section, you can specify an array of piepline selectors: Configure the Elasticsearch output | Filebeat Reference [8.8] | Elastic

I hope that helps!