Ingest Pipeline Dissect Pattern unable to match Append modifiers

[paolovalladolid]

This is is the dissect pattern

%{+dateStr} %(+dateStr) %{logLevel} %{className} %{httpNio} %{+messageContent} %{+messageContent} %{+messageContent} %{}

This is a sample line from the document that the dissect is failing on:

2023-09-11 20:43:56,987 ERROR c.i.a.c.GlobalExceptionHandler [http-nio-9095-exec-39] exception : No value present for 

This is the error message:

Unable to find match for dissect pattern: %{+dateStr} %(+dateStr) %{logLevel} %{className} %{httpNio} %{+messageContent} %{+messageContent} %{+messageContent} %{} against source: 2023-09-11 20:43:56,987 ERROR c.i.a.c.GlobalExceptionHandler [http-nio-9095-exec-39] exception : No value present for

My understanding is that ${messageContent} should have matched and appended "exception : No value present for ". However this is not the case.

[stephenb (Stephen Brown)]

Hi @paolovalladolid

You have a couple issues

1. You can not use the same variable name for different fields they will just be overridden
2. you have a typo on the 2nd date string %(+dateStr) should be %{+dateStr2}
3. You are not accounting for literal characters

Here is my example

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "dissect": {
          "field": "message",
          "pattern": "%{+dateStr} %{+dateStr2} %{logLevel} %{className} [%{httpNio}] %{+messageContent}"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "2023-09-11 20:43:56,987 ERROR c.i.a.c.GlobalExceptionHandler [http-nio-9095-exec-39] exception : No value present for"
      }
    }
  ]
}

Results

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "logLevel": "ERROR",
          "dateStr": "2023-09-11",
          "dateStr2": "20:43:56,987",
          "className": "c.i.a.c.GlobalExceptionHandler",
          "message": "2023-09-11 20:43:56,987 ERROR c.i.a.c.GlobalExceptionHandler [http-nio-9095-exec-39] exception : No value present for",
          "httpNio": "http-nio-9095-exec-39",
          "messageContent": "exception : No value present for"
        },
        "_ingest": {
          "timestamp": "2023-09-12T03:27:33.435670498Z"
        }
      }
    }
  ]
}

[leandrojmp (Leandro Pereira)]

@stephenb

• You can not use the same variable name for different fields they will just be overridden
• you have a typo on the 2nd date string %(+dateStr) should be %{+dateStr2}

Not exactly, you can use the same variable name if you use the append modifier, the + sign, as it was used in the pattern.

The issue here was basically the literals not being accounted for.

One thing, that should be added is the append_separator config, per default it will be an empty string (in logstash it is a single space).

I think this is close of what the OP wants:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "dissect": {
          "field": "message",
          "pattern": "%{+dateStr} %{+dateStr} %{logLevel} %{className} [%{httpNio}] %{+messageContent}",
          "append_separator": " "
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "2023-09-11 20:43:56,987 ERROR c.i.a.c.GlobalExceptionHandler [http-nio-9095-exec-39] exception : No value present for"
      }
    }
  ]
}

[stephenb (Stephen Brown)]

Ahhh did Not get that. I wasn't sure what he was trying to accomplish.

There was still a typo with the wrong ( versus braces { and missing literals

Thank @leandrojmp now I get it!

[paolovalladolid]

Thank you Stephen and Leandro. I had the below typed in last night but forgot to click the Reply button. I appreciate the feedback about the [ and ] literals and the append_separator.

I ended up replacing the Dissect Processor with a Grok Processor with this pattern

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel} %{JAVACLASS:javaClass} %{DATA:thread_id} %{JAVALOGMESSAGE:content}

This processed the document without issue. Our main priority is to be able to find this document by querying on the ```logLevel" field, so this is satisfied.

I was reluctant to switch to Grok but I've been getting burned lately by changes in the structure of the log text. Still, I'll keep your advice in mind if performance becomes an issue.