Cannot use example Grok Pattern (official docs) in Grok Processor

Elastic Stack Kibana
1

This is the pattern I'm trying to use in the Grok processor:

%{IPORHOST:source.ip} %{USER:user.id} %{USER:user.name} \[%{HTTPDATE:@timestamp}\] "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:int} (?:-|%{NUMBER:http.response.body.bytes:int}) %{QS:http.request.referrer} %{QS:user_agent}

The pattern is lifted straight off the official docs here:

But when I try to add the processor to my Ingest Pipeline, the pattern is highlighted red and I do not get a Save button

image
image1060×1153 49.6 KB

Same pattern works fine in Grok Debugger in Dev Tools

image
image2164×1305 119 KB

Something about HTTPDATE is breaking the processor in Ingest Pipeline. This cut down pattern still causes an issue

%{IPORHOST:source.ip} %{USER:user.id} %{USER:user.name} \[%{HTTPDATE:thetime}\]

It seems like the Grok processor does not recognize \ as an escape character when running inside Ingest Pipeline. Is there an alternative escape character or pattern?

2

We are actually on 7.17. Unfortunately I have no control over when, if ever, we will upgrade to Kibana 8.x.

I took the grok pattern from here:

It does not cause JSON format errors like the 8.x pattern. However, the processor keeps getting skipped. This is the condition:

ctx?.fields?.tomcat_app_id != null && ctx.fields.tomcat_app_id == 'ams-cache-manager-ingress'"

This is the relevant portion of the document

"tomcat_app_id": "ams-cache-manager-ingress"

This is the verbose output from the pipeline after I run it, stating the processor was skipped.

        {
          "processor_type": "grok",
          "status": "skipped",
          "if": {
            "condition": "ctx?.fields?.tomcat_app_id != null && ctx.fields.tomcat_app_id == 'ams-cache-manager-ingress'",
            "result": false
          }
        }

The document clearly shows tomcat_app_id is the correct value. I don't get why the processor would think it does not match.

3

Can you share a sample of your document that is not working?

4

I figured out the causes of the issue:

  1. Need to configure output.elasticsearch in our filebeat-kubernetes.yaml to point to the ingest pipeline, by adding a pipeline: our-pipeline statement.
  2. The YAML manifest file had fields_under_root: true. I didn't understand what that really meant until I reread the documentation. The custom field tomcat_app_id was being stored at the top level, not under a fields sub dictionary, so the conditional for the Grok processor was wrong.

Now the document is being processed as expected.

Thanks!!!