Want to match against single grok pattern and multiple patterns in same filter

Elastic Stack Logstash
1

I want to have below kinds of match patterns in my logstash configuration file to match different lines.

(a) match => { "message" => "\[%{TIMESTAMP_ISO8601:bip_timestamp}\] \[%{WORD:app_server}] \[%{WORD:log_level}] \[] \[%{DATA:thread_type}\] \[tid: %{NUMBER:tid}\] \[userId: %{DATA:userId}\] (?<event_type>\sstart-getchunk): Report=(?<report_name>[^,]*), TemplateName=(?<template_name>[^,]*), OutFormat=(?<output_format>[^,]*), Locale=(?<locale>[^,]*)" }

(b) match => { "message" => "\[%{TIMESTAMP_ISO8601:bip_timestamp}\] \[%{WORD:app_server}] \[%{WORD:log_level}] \[] \[%{DATA:thread_type}\] \[tid: %{NUMBER:tid}\] \[userId: %{DATA:userId}\] (?<event_type>\send-getchunk): Report=(?<report_name>[^,]*), TemplateName=(?<template_name>[^,]*), OutFormat=(?<output_format>[^,]*), Locale=(?<locale>[^,]*)" }

(c) to match a multiline having this match break_on_match => false match => { "message => [ "Report path: (?<report_path>[^,\r\n]*)", "Datamodel name: (?<datamodel_name>[^,\r\n]*)", "User name: (?<user_name>[^,\r\n]*)", ] }

(d) other n number of matches like the one mentioned in points (a) & (b) above

For now, let's assume I have only (a), (b) & (c) in my config.

My config looks similar to below (with the sequence of "match =>" lines changed).

filter{
	grok{
		break_on_match => false
		match => { 	"message" => "\[%{TIMESTAMP_ISO8601:bip_timestamp}\] \[%{WORD:app_server}] \[%{WORD:log_level}] \[] \[%{DATA:thread_type}\] \[tid: %{NUMBER:tid}\] \[userId: %{DATA:userId}\] (?<event_type>\sstart-getchunk): Report=(?<report_name>[^,]*), TemplateName=(?<template_name>[^,]*), OutFormat=(?<output_format>[^,]*), Locale=(?<locale>[^,]*)" }
		match => { 	"message" => "\[%{TIMESTAMP_ISO8601:bip_timestamp}\] \[%{WORD:app_server}] \[%{WORD:log_level}] \[] \[%{DATA:thread_type}\] \[tid: %{NUMBER:tid}\] \[userId: %{DATA:userId}\] (?<event_type>\send-getchunk): Report=(?<report_name>[^,]*), TemplateName=(?<template_name>[^,]*), OutFormat=(?<output_format>[^,]*), Locale=(?<locale>[^,]*)" }
		match => {
			"message => [
				"Report path: (?<report_path>[^,\r\n]*)",
				"Datamodel name: (?<datamodel_name>[^,\r\n]*)",
				"User name: (?<user_name>[^,\r\n]*)",
			]
		}
	}
}

Now the problem statement: -
The config matches pattern of only THE LAST of the three "match =>" mentioned in configuration file.
So, if in config file
[A] the sequence of "match =>" lines are (a) -> (b) -> (c) (as shown in above config): it matches patterns for only (c).
[B] the sequence of "match =>" lines are (c) -> (b) -> (a): it matches patterns for only (a).
[C] the sequence of "match =>" lines are (c) -> (a) -> (b): it matches patterns for only (b).

But, I want all (a), (b), (c) to be matched in the input.

Also, I want to maintain the match patterns for (a), (b), (c) as it is.
Actually, for (c) I had a grok pattern like (a) and (b). At that time the config was trying to match all of (a), (b) and (c). But becasue of "_groktimeout" error for (c), changed the pattern to the one mentioned in (c) here [Ref].

To solve the issue, I tried having

  • multiple grok{} blocks under same filter{} block AND
  • multiple filter{} blocks in the same config file
    to solve the issue by splitting "(a), (b)" and "(c)" in separate blocks. But in the above two cases, it does not generate any output.
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.