Grok Pattern to extract brackets content

Elastic Stack Logstash
1

Hi , I am attempting to create a grok pattern to parse the following log file.

14:10:49:158017|5860-00088:JMIX: ChainSubscription {INFO} Action {Update} Chain
{MOVE.AA_MOVE_TOP.ABC_SWITCH_COMP_GO.SHOP_ABC} Snapshot {false}

I would like to assign fields in Kibana to;

ChainSubscription = INFO
Action= Update
Chain= MOVE.AA_MOVE_TOP.ABC_SWITCH_COMP_GO.SHOP_ABC
Snapshot= false

appreciate any help here.

2

Are you always going to have exactly those keys, or are the keys variable too?

If the keys are variable, the kv filter may be helpful:

filter {
  # first, split the message into component parts. I don't know
  # what the format of yours means, so I used the dissect filter
  # to split on the `:JMIX:` sequence. You can use grok if you'd
  # like to achieve similar. The point though, is that the entire
  # key/value sequence is put in a single var `[@metadata][kv]`.
  dissect {
     mapping => {
        "message" => "%{a}:JMIX:%{[@metadata][kv]}"
      }
  }
  # now we use the KV filter to split that up. This won't work well
  # if your squiggle-bracket-quoted values contain squiggle-brackets,
  # since it has no way to differentiate between a _meaningful_
  # squiggle-bracket and a _literal_ one.
  kv {
    "source" => "[@metadata][kv]"
    "field_split" => "}"
    "value_split" => "{"
    "trim_key" => " "
    "trim_value" => " "
  }
}
3

thank you

I couldn't get it to work. I am still working on your solution.

using this seems to work alot better.
filter {
grok {
match => { "message" => ["%{TIME}|%{WORD:info}-%{WORD:ID2}:%{WORD:mgs_level}:%{SPACE}%{WORD:code}%{SPACE}{%{W ORD:data2}}%{SPACE}%{WORD:code3}%{SPACE}{%{WORD:data3}}%{SPACE}%{WORD:code4}%{SPACE}{(?.*?)}%{GREEDYDATA:msg}" ] }
}
}

4

Grok patterns can get pretty complex; I prefer to start with the Dissect filter, and only move on to Grok if I encounter input that cannot be handled by Dissect. Dissect is especially great because you don't have to define perfect patterns for each captured variable, so it's easier to get things right.

Here, I've used your names for things and applied it to a Dissect matcher.

filter {
  # first, split the message into component parts. I don't know
  # what the format of yours means, so I used the dissect filter
  # to split on the `:JMIX:` sequence. You can use grok if you'd
  # like to achieve similar. The point though, is that the entire
  # key/value sequence is put in a single var `[@metadata][kv]`.
  dissect {
     mapping => {
        "message" => "%{}|%{info}-%{ID2}:%{mgs_level}: %{[@metadata][kv]}"
      }
  }
  # now we use the KV filter to split that up. This won't work well
  # if your squiggle-bracket-quoted values contain squiggle-brackets,
  # since it has no way to differentiate between a _meaningful_
  # squiggle-bracket and a _literal_ one.
  kv {
    "source" => "[@metadata][kv]"
    "field_split" => "}"
    "value_split" => "{"
    "trim_key" => " "
    "trim_value" => " "
  }
}

With your input, the rubydebug output looks like:

{
                  "ID2" => "00088",
            "mgs_level" => "JMIX",
             "Snapshot" => "false",
                 "info" => "5860",
    "ChainSubscription" => "INFO",
               "Action" => "Update",
                "Chain" => "MOVE.AA_MOVE_TOP.ABC_SWITCH_COMP_GO.SHOP_ABC",
              "message" => "14:10:49:158017|5860-00088:JMIX: ChainSubscription {INFO} Action {Update} Chain {MOVE.AA_MOVE_TOP.ABC_SWITCH_COMP_GO.SHOP_ABC} Snapshot {false}",
             "@version" => "1",
                 "host" => "castrovel.local",
           "@timestamp" => 2018-09-26T17:17:50.758Z
}
5

many thanks

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.