Error when filter multiline of rabbitmq log

Elastic Stack Logstash
1

When i parse log from rabbitmq, i got the error as below:
`

[2017-10-12T04:01:18,006][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"The setting type in plugin multiline is obsolete and is no longer available. You can achieve this same behavior with the new conditionals, like: if [type] == \"sometype\" { multiline { ... } }. If you have any questions about this, you are invited to visit Logstash - Discuss the Elastic Stack and ask."}

`

The following is my config:

input {
	file {
		type => "rabbit"
		path => "/home/ubuntu/logstash-5.6.2/rabbitmq.log"
	}
}
filter {
	multiline{
		type => "rabbit"
		pattern => "^="
		negate => true
		what => "previous"
	}
	grok {
		type => "rabbit"
		patterns_dir => "patterns"
		pattern => "^=%{WORD:report_type} REPORT=+ %{RABBIT_TIME:time_text} ===.*$"
	}
	mutate {
		type => "rabbit"
		add_field => [ "message", "%{@message}" ]
	}
	mutate {
		gsub => [
		  "message", "^=[A-Za-z0-9: =-]+=\n", "",
		  # interpret message header text as "severity"
		  "report_type", "INFO", "1",
		  "report_type", "WARNING", "3",
		  "report_type", "ERROR", "4",
		  "report_type", "CRASH", "5",
		  "report_type", "SUPERVISOR", "5"
		]
	}
	
}
output {
	stdout { codec => rubydebug }
}

Could someone help me in this case?

2

The multiline filter plugin has been deprecated. Instead use the multiline codec with your file input plugin.

3

So how can i use logstash filter to parse rabbitmq log?

4

Specify a multiline codec for the file input plugin and remove the multiline filter. You should be able to keep the rest the same, although I do not understand why you are specifying type => "rabbit" for the filter plugins.

5

I edited as your recommend. The following is my edition:

input {
	file {
		type => "rabbit"
		path => "/home/ubuntu/logstash-5.6.2/rabbitmq.log"
		codec => multiline{
			pattern => "^="
			negate => true
			what => "previous"
		}
	}
}
filter {

grok {
	type => "rabbit"
	patterns_dir => "patterns"
	pattern => "^=%{WORD:report_type} REPORT=+ %{RABBIT_TIME:time_text} ===.*$"
}
mutate {
	type => "rabbit"
	add_field => [ "message", "%{@message}" ]
}
mutate {
	gsub => [
	  "message", "^=[A-Za-z0-9: =-]+=\n", "",
	  # interpret message header text as "severity"
	  "report_type", "INFO", "1",
	  "report_type", "WARNING", "3",
	  "report_type", "ERROR", "4",
	  "report_type", "CRASH", "5",
	  "report_type", "SUPERVISOR", "5"
	]
}

}
output {
	stdout { codec => rubydebug }
}

I got the error:
[2017-10-12T07:03:20,501][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"The setting type in plugin grok is obsolete and is no longer available. You can achieve this same behavior with the new conditionals, like: if [type] == \"sometype\" { grok { ... } }. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask."}

6

You should remove this from your plugin config as I do not think it is supported any longer. You can see exactly which configuration parameters that are supported in the documentation.

7

I changed my config to your recommend, so it still got error. Could you give me some suggests or examples to parse rabbitmq log using logstash filter?
Thank you.

8

What does the config look like now? What error are you getting?

9 
input {
	file {
		path => "/home/ubuntu/logstash-5.6.2/rabbitmq.log"
		codec => multiline{
			pattern => "^="
			negate => true
			what => "previous"
		}
	}
}
filter {
	
	grok {
		patterns_dir => "patterns"
		match => {"message" => "^=%{WORD:report_type} REPORT=+ %{RABBIT_TIME:time_text} ===.*$"}
	}
	mutate {
		add_field => [ "message", "%{@message}" ]
	}
	mutate {
		gsub => [
		  "message", "^=[A-Za-z0-9: =-]+=\n", "",
		  # interpret message header text as "severity"
		  "report_type", "INFO", "1",
		  "report_type", "WARNING", "3",
		  "report_type", "ERROR", "4",
		  "report_type", "CRASH", "5",
		  "report_type", "SUPERVISOR", "5"
		]
	}
	
}
output {
	stdout { codec => rubydebug }
}

The error:
[2017-10-12T07:20:53,132][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<Grok::PatternError: pattern %{RABBIT_TIME:time_text} not defined>, :backtrace=>["/home/ubuntu/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in `compile'",...........

10

As the error message says it's not able to find a definition of your RABBIT_TIME pattern. I suggest you use the absolute path in the grok filter's patterns_dir option.

11

Thanks magnusbaeck, there are no error when i change config as recommend. However, i can't get any result on screen when read log from file as input.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.