GROK filter throws ERROR

Elastic Stack Logstash
1

I have written a filter to parse logs based on some condition :

filter {

	if [message] =~ "\tat" {
		grok {
		  match => ["message", "^(\tat)"]
		  add_tag => ["stacktrace"]
		}
    }
  
  grok {

	  if([fields][log_type] == "tomcat" || [fields][log_type] == "wildfy") {

		match => [
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{JAVACLASS:class}\] (\(%{DATA:thread}\) )?%{GREEDYDATA:logMessage}$', #spring logs
			"message",'^%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)', #tomcat logs		
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\s+\[(?<package>[a-z.]*)\] \(%{DATA:thread}\) (?<WILDFY_CODE>[A-Z0-9]*): %{GREEDYDATA:log}$' #wildfy
		]		
	  }
	  
	  else {
		match => [
				"message",'%{GREEDYDATA:walla}'
		]	
	  }
    }
}

and the exception that I see when I start logstash is :

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 21, column 6 (byte 174) after filter {\n\n\tif [message] =~ \"\\tat\" {\n\t\tgrok {\n\t\t match => [\"message\", \"^(\\tat)\"]\n\t\t add_tag => [\"stacktrace\"]\n\t\t}\n }\n\n grok {\n\n\t if", :backtrace=>["C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:49:in initialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:167:ininitialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/agent.rb:305:inblock in converge_state'"]}`

2

You can not have conditionals within a filter definition. Move the conditionals outside the grok block and instead specify multiple grok blocks.

3

Can you please correct the snippet if in error ?

filter {
    	if [message] =~ "\tat" {
    		grok {
    		  match => ["message", "^(\tat)"]
    		  add_tag => ["stacktrace"]
    		}
        }
	
	if ([fields][log_type] == "tomcat") {
    mutate {
      replace => {
        "[type]" => "tomcat"
      }
    }
  }
  else if ([fields][log_type] == "wildfy") {
    mutate {
      replace => {
        "[type]" => "wildfy"
      }
    }
  }
  
  grok {

	  if(%{type} == "tomcat" || %{type} == "wildfy") {

		match => [
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{JAVACLASS:class}\] (\(%{DATA:thread}\) )?%{GREEDYDATA:logMessage}$', #spring logs
			"message",'^%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)', #tomcat logs		
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\s+\[(?<package>[a-z.]*)\] \(%{DATA:thread}\) (?<WILDFY_CODE>[A-Z0-9]*): %{GREEDYDATA:log}$' #wildfy
		]		
	  }
	  
	  else {
		match => [
				"message",'%{GREEDYDATA:walla}'
		]	
	  }
    }
}
4

Apart from the problem Chrstian mentioned,

if(%{type} == "tomcat" || %{type} == "wildfy") {

is wrong. Change to

  if [type] == "tomcat" or [type] == "wildfy" {

or

  if [type] in ["tomcat", "wildfy"] {
1 Like
5

As you have suggest in Multiline issue with 2 different patterns for a single event I have put :

beats {
	port=>5044
	multiline {
	  pattern => "^%{TIMESTAMP_ISO8601}"
	  negate => true
	  what => "previous"
	}
}

and in the beginning of filter before anything I have put :

overwrite => ["message"]

should this configuration work for parsing.

6

If you are getting data from Filebeat you should consider performing the multiline processing there instead. You always want to do this as close to the source as possible.

7 
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:/data/logs_server/apache-tomcat-8.0.39.log
  fields: {log_type: tomcat}
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

I am using the above pattern in file beat but logstash is throwing error can you please suggest ?

8

I am using the above pattern in file beat but logstash is throwing error can you please suggest ?

Without seeing the Logstash error we can't help out.

9

I have posted it in my original post, in the beginning itself.

10

Oh, that error. As we've said put conditionals outside the grok filter.

if ... {
  grok { ... }
} else {
  grok { ... }
}
1 Like
11

With all the updates I am getting the following exception:

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 16, column 12 (byte 160) after filter {\n\n\toverwrite ", :backtrace=>["C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-08T12:51:30,487][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

So my filebeat config is

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:/data/logs_server/apache-tomcat-8.0.39.log
  fields: {log_type: tomcat}
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

My logstash config is

input {

beats {
	port=>5044
	codec => multiline {
	  pattern => "^%{TIMESTAMP_ISO8601}"
	  negate => true
	  what => "previous"
	}
}

}

filter {

	overwrite => ["message"]

	if [message] =~ "\tat" {
		grok {
		  match => ["message", "^(\tat)"]
		  add_tag => ["stacktrace"]
		}
    }
	
	if ([fields][log_type] == "tomcat") {
    mutate {
      replace => {
        "[type]" => "tomcat"
      }
    }
  }
  else if ([fields][log_type] == "wildfy") {
    mutate {
      replace => {
        "[type]" => "wildfy"
      }
    }
  }
  
  if [type] in ["tomcat", "wildfy"] {
	  grok {  

			match => [
				"message",'^%{TIMESTAMP_ISO8601:betimestamp} %{LOGLEVEL:logLevel}%{SPACE}\[%{JAVACLASS:className}\] (\(%{DATA:threadName}\) )?%{GREEDYDATA:logMessage}$'
			]
		}
	} else {
		  grok {
			match => [
					"message",'%{GREEDYDATA:unparsedText}'
			]	
		  }
		}	 
}

Please suggest.

12

overwrite => ["message"]

Remove.

13

Done. Am using 6.3.x version of ELK stack. Still the exception is there but has changed as :

[ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Beats port=>5044, codec=><LogStash::Codecs::Multiline pattern=>\"^%{TIMESTAMP_ISO8601}\", negate=>true, what=>\"previous\", id=>\"a1da3fb1-8767-44c0-a30d-19230430f5ae\", enable_metric=>true, charset=>\"UTF-8\", multiline_tag=>\"multiline\", max_lines=>500, max_bytes=>10485760>, id=>\"38208dd42da36e7ce6566aa40fac509ec2c67190e9b450a69ed80be203672a0e\", enable_metric=>true, host=>\"0.0.0.0\", ssl=>false, ssl_verify_mode=>\"none\", include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>[\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256\", \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256\"], client_inactivity_timeout=>60, executor_threads=>4>", :error=>"Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html", :thread=>"#<Thread:0x24284407 run>"}
[ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html>, :backtrace=>["C:/tools/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.14-java/lib/logstash/inputs/beats.rb:153:in register'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:340:inregister_plugin'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in register_plugins'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:498:instart_inputs'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:392:in start_workers'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:288:inrun'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=>"#<Thread:0x24284407 run>"}
[2018-08-08T14:38:46,039][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

14

But the error message tells you in plain English what the problem is and links to the relevant documentation.

15

And that's what I am also using if you see my filebeat configuration file.

16

Then delete the multiline codec in your Logstash configuration.

17

yes done that, but how to add all my error stacktraces as a single group ?

18

That's what a proper multiline configuration should take care of. I don't know what your logs look like so I can't make a concrete suggestion, but the Timestamps example from the documentation is probably what you're looking for.

19

The exception stacktrace is a typical JAVA exception. As a sample I can have :

2018-08-03 16:55:06.470 [ThreadPoolTaskScheduler-4] ERROR  o.s.s.s.TaskUtils$LoggingErrorHandler.handleError - Unexpected error occurred in scheduled task.
com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
	at com.mongodb.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:477)
	at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:228)
	at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:96)
	...
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	...
	at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:224)
	... 31 common frames omitted

or

2018-08-07 15:26:02.404 [DefaultQuartzScheduler_Worker-1] ERROR org.slf4j.helpers.SubstituteLogger.error - Unable to start/stop JMX
javax.management.InstanceAlreadyExistsException: com.jolbox.bonecp:type=BoneCP
	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
	...
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
20

Okay, so a pretty standard log. Follow the Timestamps example (minor adjustments might be necessary).