Ingest pipeline grok processor on_failure

Elastic Stack Elasticsearch
1

Hi,
I have an elastic ingest pipeline with grok processor defined along with error handling

{
  "my_ingest" : {
    "description" : "parse multiple patterns",
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            """^\[end  ] %{DATA:method} \'%{GREEDYDATA:url}' %{DATA:status} :: Duration: %{DATA:duration} ms""",
            """^\[start] %{DATA:method} \'%{GREEDYDATA:url}' :: Start Time:%{GREEDYDATA:starttime}""",
            "%{GREEDYDATA:message}"
          ]
        }
      }
    ],
    "on_failure" : [
      {
        "set" : {
          "field" : "_index",
          "value" : "failed-{{ _index }}"
        }
      }
    ]
  }
}

i am referring to this pipeline in my filebeat.yml
the grok filters work when i do a simulate. in dev tools. But when i run the actual logging i do not see the log statements. it looks like they are failing to get parsed and not visible in kibana. i also don't see a new index created where i am hoping to see the errors logged as defined on on_failure.
can some one please suggest or give pointers for debugging the issue.

thanks

2

Hi,
how do i access this on_failure_processor_type and on_failure_message from the metadata ?

"on_failure" : [
          {
            "set" : {
              "field" : "error",
              "value" : "{{_ingest.on_failure_processor_type }}-{{ _ingest.on_failure_message }}"
            }
          }
        ]

Thanks

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.