Dissect Filter Special Characters

[AO-StreetArt (Ao)]

I am trying to use dissect to break apart a multi-line string which contains the characters '{' and '}' in what will be a delimiter. To do this, I would need to use escape characters (such as '\n') within the dissect filter string. Is this supported?

[guyboertje]

Sorry, this is not supported.

What does your sample data look like? Maybe I can suggest something.

[AO-StreetArt (Ao)]

Thank you Guy, here's a sample. The full line is actually quite a bit larger, complex enough that Grok filters time out when attempting to parse it:

DIRECT-ROUTE[1556],[1534]{ABC123->DEF456}[1193(SH-50793847)]{GHI789->} CombinedRelayLegIndex: -1

    Total Reduced Cost: 321.539978, Partial Reduced Cost: 321.539978

[guyboertje]

Please show me where your expected fields are, I don't understand. Maybe use caret characters.

e.g.

DIRECT-ROUTE[^1556^],[^1534^]{^ABC123->DEF456^}[^1193(SH-50793847)^]{^GHI789->^}

[AO-StreetArt (Ao)]

My apologies, here is the current dissect filter I have currently:

%{ROUTE_TYPE}-ROUTE[%{ROUTE_ID}],%{ROUTE_DATA}

This is successful in parsing the first two needed fields. However, I also need to pull out, for example, the text within the '{' and '}' characters (ie. the field after running the filter should contain the text 'ABC123->DEF456'.

Then, I need to split the number values from the second line (ie. the field after running the filter should contain the text '321.539978').

Currently, I'm stuck simply passing the majority of the line into the final variable, as I can't figure out how to represent the '{' character or the '\n' within a delimiter.

I'm not able to find anything about the caret character in the dissect documentation, what effect does it have?

Thank you for your assistance!

[guyboertje]

Sorry about the caret misunderstanding - I meant - use it here for illustration purposes to show me where the different field start and end as per the example. Ignore this.

this works...

input {
  generator {
    lines => [
      "DIRECT-ROUTE[1556],[1534]{ABC123->DEF456}[1193(SH-50793847)]{GHI789->} CombinedRelayLegIndex: -1

    Total Reduced Cost: 321.539978, Partial Reduced Cost: 321.539978"
    ]
    # time => 300
    count => 1
  }
}

filter {
  dissect {
    mapping => {
      message => "%{route_type}-ROUTE[%{route_id}],[%{?skip1}]%{route_detail}[%{?skip2}Total Reduced Cost: %{trc}, Partial Reduced Cost: %{prc}"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

event looks like...

{
             "prc" => "321.539978",
        "sequence" => 0,
      "@timestamp" => 2017-09-08T16:42:44.585Z,
        "route_id" => "1556",
             "trc" => "321.539978",
      "route_type" => "DIRECT",
        "@version" => "1",
            "host" => "Elastics-MacBook-Pro.local",
         "message" => "DIRECT-ROUTE[1556],[1534]{ABC123->DEF456}[1193(SH-50793847)]{GHI789->} CombinedRelayLegIndex: -1\n\n    Total Reduced Cost: 321.539978, Partial Reduced Cost: 321.539978",
    "route_detail" => "{ABC123->DEF456}"
}

NOTES:
There is a bug (fixed but not released) in the regex that detects the starting %{ and the ending }. This means that you should use mutate gsub later to remove the { and } in the field I called `route_detail. When the bug fix is released this will not be necessary.
Jump over the newlines with a skip field, The newlines do not need to be specified in the dissect mapping string.

[AO-StreetArt (Ao)]

Brilliant! This works perfectly! Thank you!