Unable to use grok pattern on GREEDYDATA message

Elastic Stack Logstash
1

Hi,

I have this log message -
2022-03-08 04:16:04 [DEBUG] Creating linked clone: from Template-CentOS, to CentOS-001122

My logstash config. file is as follows -

filter {
    grok {
      match => {
        "message" => [
          "^%{TIMESTAMP_ISO8601:logTimestamp} \[%{LOGLEVEL:logLevel}\] %{DATA} %{DATA} %{DATA:actionName} %{GREEDYDATA:logMessage}$"
        ]
      }
    }

    if [actionName] == "clone:" {
      grok {
        match => {
          "logMessage" => [
            "^%{TIMESTAMP_ISO8601:logTimestamp} \[%{LOGLEVEL:logLevel}\] %{DATA} %{DATA} %{DATA:actionName} %{WORD} %{DATA:templateName} %{GREEDYDATA:logMessage}$"
          ]
        }
      }
    }
}

My actionName field contains string "clone:" and I have verified that it is entering the if loop using mutate filter (remove and rename fields are working). The only time I am having issue is when I am looking to grok for template name in the above copied log i.e. want to retrieve "Template-CentOS" name.

I have reviewed a lot of KB articles, however, couldn't get through. Can someone please point out what am I doing wrong?

Thanks,

2

Not quite sure your question but if all your logs follow a similar format then this should work.

^%{TIMESTAMP_ISO8601:logTimestamp} \[%{LOGLEVEL:logLevel}\] Creating linked %{DATA:actionName}: from %{DATA:from}\, to %{GREEDYDATA:to}

Output

{
  "logTimestamp": "2022-03-08 04:16:04",
  "logLevel": "DEBUG",
  "from": "Template-CentOS",
  "to": "CentOS-001122",
  "actionName": "clone"
}
1 Like
3

Hi Aaron,

Thanks a lot for your response. Now that's where the challenge lies since there are several different events being logged in the files.

For example:

2022-03-28 07:05:31 [ERROR] save_variable: Error while trying to work with DB. Exception => (1205, u'Lock wait timeout exceeded; try restarting transaction')
2022-03-25 12:37:11: [DEBUG] diag-_do_action [1] action_name='alert' action_params='['admins', 'N5K failed to boot up']' ### dlgscr.py: 382: _do_action(): 
2022-03-08 04:16:04 [DEBUG] Creating linked clone: from Template-CentOS, to CentOS-001122

Regards,
Rahul

4

In addition to my previous comment, I am parsing different kind of events from logs. And I do segregating with different field names in Kibana visualisations.

For example:

2022-03-25 12:37:11: [DEBUG] diag-_do_action [1] action_name='alert' action_params='['admins', 'N5K failed to boot up']' ### dlgscr.py: 382: _do_action():

I am creating an alert visualisation in Kibana for the above event.

2022-03-08 04:16:04 [DEBUG] Creating linked clone: from Template-CentOS, to CentOS-001122

I am creating a template name visualisation for above event.

Note: The source of all these events is same.

Thanks,

5

If you want to parse all the different types of messages then you need to write grok patterns for all of them. It's not possible to write 1 to parse to the level you want.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.