Logstash dissect and grok pattern matching issue

Elastic Stack Logstash
1

Hi, I'm trying to use dissect and grok pattern matching together in order to get the desired field:value pair that I want. However, I don't understand why it's not working.

I'm parsing the description field which can come in multiple ways like this:

"description"=>"SERVERNAME.TC.AT;LOCATION COLLECTOR"
"description"=>"SERVERNAME"
"description"=>"WORDS WITH SPACES IN BETWEEN THEM"
"description"=>"SERVERNAME.TC.AT
"description"=>"TEXT;DELIMITED;BY;SEMICOLONS"

I want to target the last description pattern with the "text;delimited;by;semicolons" pattern so I'm using dissect to grab the second value from the delimited string.

But as you can see from above, the description comes in a few different formats. So for those other formats, I just want to use grok and store the whole string as-is into my description field.

Here is what Logstash data looks like:

if [description] {
  dissect {
    mapping => {"description" => "%{};%{node-name};%{};%{};%{}"}
  }
  if ("_dissectfailure" in [tags]) {
        grok {
            match => {"message" => "%{(?<description>.+)}"}
            remove_field => [ "tags" ]
        }
  }
}

I keep receiving _dissect and _grokparsefailures and not sure why.

Thank you.

2

The remove_field option on the grok filter is only applied if the grok filter matches, otherwise a _grokparsefailure tag is added. The grok will only match if your [message] field contains %{ and }. I think you meant

match => {"message" => "(?<description>.+)"}

but to do that it would be cheaper to use mutate

mutate {
    add_field => { "description" => "%{message}" }
    remove_field => [ "tags" ]
}
3

Yes, that is what I meant. Couldn't figure out when or when not to use {}.

I also noticed a mistake with my initial grok filter. I was matching on a field called "message" when it should have been "description". This appers to be working now:

if ("_dissectfailure" in [tags]) {
    grok {
        match => {"description" => "(?<description>.+)"}
        remove_field => [ "tags" ]
    }
}

I believe your recommendation was to do this:

if ("_dissectfailure" in [tags]) {
    mutate {
        add_field => { "description" => "%{description}" }
        remove_field => [ "tags" ]
    }
}

However, that was taking the contents of the description field and appending it to itself. Basically duplicating it.

Ex: I was seeing this in the JSON output in Kibana

"description": [
    "NODE1.RD.OM",
    "NODE1.RD.OM"
],

Is that what is supposed to happen?

4

Yes, if a field already exists add_field will turn it into an array.

But match => {"description" => "(?<description>.+)"} does not make any sense. It just sets the description field to have the value of the description field. It is a no-op. Perhaps replace

dissect {
    mapping => {"description" => "%{};%{node-name};%{};%{};%{}"}
}
if ("_dissectfailure" in [tags]) {
    grok {
        match => {"message" => "%{(?<description>.+)}"}
        remove_field => [ "tags" ]
    }
}

with

dissect {
    mapping => {"description" => "%{};%{node-name};%{};%{};%{}"}
    tag_on_failure => []
}
5

That was my intention, so that I stop seeing the _dissectfailure tags and also to stop seeing the "Dissector mapping, pattern not found" [WARN] messages in the logs.

When I implement this, the _dissectfailure tags go away but the [WARN] messages are still logging.

6

Yes, the logging is unconditional.

What you can do is

if [description] =~ /(.+;){4}.+/ {
    dissect { ...
7

Thanks, the regex magic works like a charm. Much cleaner looking too. :slightly_smiling_face:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.