GROK filter throws ERROR

Raghuveer_SJ · August 8, 2018, 5:04am

I have written a filter to parse logs based on some condition :

filter {

	if [message] =~ "\tat" {
		grok {
		  match => ["message", "^(\tat)"]
		  add_tag => ["stacktrace"]
		}
    }
  
  grok {

	  if([fields][log_type] == "tomcat" || [fields][log_type] == "wildfy") {

		match => [
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{JAVACLASS:class}\] (\(%{DATA:thread}\) )?%{GREEDYDATA:logMessage}$', #spring logs
			"message",'^%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)', #tomcat logs		
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\s+\[(?<package>[a-z.]*)\] \(%{DATA:thread}\) (?<WILDFY_CODE>[A-Z0-9]*): %{GREEDYDATA:log}$' #wildfy
		]		
	  }
	  
	  else {
		match => [
				"message",'%{GREEDYDATA:walla}'
		]	
	  }
    }
}

and the exception that I see when I start logstash is :

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 21, column 6 (byte 174) after filter {\n\n\tif [message] =~ \"\\tat\" {\n\t\tgrok {\n\t\t match => [\"message\", \"^(\\tat)\"]\n\t\t add_tag => [\"stacktrace\"]\n\t\t}\n }\n\n grok {\n\n\t if", :backtrace=>["C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:49:in initialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:167:ininitialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/agent.rb:305:inblock in converge_state'"]}`

Christian_Dahlqvist · August 8, 2018, 5:25am

You can not have conditionals within a filter definition. Move the conditionals outside the grok block and instead specify multiple grok blocks.

Raghuveer_SJ · August 8, 2018, 5:40am

Can you please correct the snippet if in error ?

filter {
    	if [message] =~ "\tat" {
    		grok {
    		  match => ["message", "^(\tat)"]
    		  add_tag => ["stacktrace"]
    		}
        }
	
	if ([fields][log_type] == "tomcat") {
    mutate {
      replace => {
        "[type]" => "tomcat"
      }
    }
  }
  else if ([fields][log_type] == "wildfy") {
    mutate {
      replace => {
        "[type]" => "wildfy"
      }
    }
  }
  
  grok {

	  if(%{type} == "tomcat" || %{type} == "wildfy") {

		match => [
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{JAVACLASS:class}\] (\(%{DATA:thread}\) )?%{GREEDYDATA:logMessage}$', #spring logs
			"message",'^%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)', #tomcat logs		
			"message",'^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\s+\[(?<package>[a-z.]*)\] \(%{DATA:thread}\) (?<WILDFY_CODE>[A-Z0-9]*): %{GREEDYDATA:log}$' #wildfy
		]		
	  }
	  
	  else {
		match => [
				"message",'%{GREEDYDATA:walla}'
		]	
	  }
    }
}

magnusbaeck · August 8, 2018, 5:53am

Apart from the problem Chrstian mentioned,

if(%{type} == "tomcat" || %{type} == "wildfy") {

is wrong. Change to

  if [type] == "tomcat" or [type] == "wildfy" {

or

  if [type] in ["tomcat", "wildfy"] {

Raghuveer_SJ · August 8, 2018, 6:39am

As you have suggest in Multiline issue with 2 different patterns for a single event I have put :

beats {
	port=>5044
	multiline {
	  pattern => "^%{TIMESTAMP_ISO8601}"
	  negate => true
	  what => "previous"
	}
}

and in the beginning of filter before anything I have put :

overwrite => ["message"]

should this configuration work for parsing.

Christian_Dahlqvist · August 8, 2018, 6:55am

If you are getting data from Filebeat you should consider performing the multiline processing there instead. You always want to do this as close to the source as possible.

Raghuveer_SJ · August 8, 2018, 7:03am

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:/data/logs_server/apache-tomcat-8.0.39.log
  fields: {log_type: tomcat}
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

I am using the above pattern in file beat but logstash is throwing error can you please suggest ?

magnusbaeck · August 8, 2018, 7:07am

I am using the above pattern in file beat but logstash is throwing error can you please suggest ?

Without seeing the Logstash error we can't help out.

Raghuveer_SJ · August 8, 2018, 7:08am

I have posted it in my original post, in the beginning itself.

magnusbaeck · August 8, 2018, 7:15am

Oh, that error. As we've said put conditionals outside the grok filter.

if ... {
  grok { ... }
} else {
  grok { ... }
}

Raghuveer_SJ · August 8, 2018, 7:17am

With all the updates I am getting the following exception:

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 16, column 12 (byte 160) after filter {\n\n\toverwrite ", :backtrace=>["C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-08T12:51:30,487][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

So my filebeat config is

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:/data/logs_server/apache-tomcat-8.0.39.log
  fields: {log_type: tomcat}
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

My logstash config is

input {

beats {
	port=>5044
	codec => multiline {
	  pattern => "^%{TIMESTAMP_ISO8601}"
	  negate => true
	  what => "previous"
	}
}

}

filter {

	overwrite => ["message"]

	if [message] =~ "\tat" {
		grok {
		  match => ["message", "^(\tat)"]
		  add_tag => ["stacktrace"]
		}
    }
	
	if ([fields][log_type] == "tomcat") {
    mutate {
      replace => {
        "[type]" => "tomcat"
      }
    }
  }
  else if ([fields][log_type] == "wildfy") {
    mutate {
      replace => {
        "[type]" => "wildfy"
      }
    }
  }
  
  if [type] in ["tomcat", "wildfy"] {
	  grok {  

			match => [
				"message",'^%{TIMESTAMP_ISO8601:betimestamp} %{LOGLEVEL:logLevel}%{SPACE}\[%{JAVACLASS:className}\] (\(%{DATA:threadName}\) )?%{GREEDYDATA:logMessage}$'
			]
		}
	} else {
		  grok {
			match => [
					"message",'%{GREEDYDATA:unparsedText}'
			]	
		  }
		}	 
}

Please suggest.

magnusbaeck · August 8, 2018, 7:46am

overwrite => ["message"]

Remove.

Raghuveer_SJ · August 8, 2018, 8:58am

Done. Am using 6.3.x version of ELK stack. Still the exception is there but has changed as :

[ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Beats port=>5044, codec=><LogStash::Codecs::Multiline pattern=>\"^%{TIMESTAMP_ISO8601}\", negate=>true, what=>\"previous\", id=>\"a1da3fb1-8767-44c0-a30d-19230430f5ae\", enable_metric=>true, charset=>\"UTF-8\", multiline_tag=>\"multiline\", max_lines=>500, max_bytes=>10485760>, id=>\"38208dd42da36e7ce6566aa40fac509ec2c67190e9b450a69ed80be203672a0e\", enable_metric=>true, host=>\"0.0.0.0\", ssl=>false, ssl_verify_mode=>\"none\", include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>[\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256\", \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256\"], client_inactivity_timeout=>60, executor_threads=>4>", :error=>"Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html", :thread=>"#<Thread:0x24284407 run>"}
[ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html>, :backtrace=>["C:/tools/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.14-java/lib/logstash/inputs/beats.rb:153:in register'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:340:inregister_plugin'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in register_plugins'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:498:instart_inputs'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:392:in start_workers'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:288:inrun'", "C:/tools/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=>"#<Thread:0x24284407 run>"}
[2018-08-08T14:38:46,039][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

magnusbaeck · August 8, 2018, 9:45am

But the error message tells you in plain English what the problem is and links to the relevant documentation.

Raghuveer_SJ · August 8, 2018, 9:47am

And that's what I am also using if you see my filebeat configuration file.

magnusbaeck · August 8, 2018, 9:48am

Then delete the multiline codec in your Logstash configuration.

Raghuveer_SJ · August 8, 2018, 10:17am

yes done that, but how to add all my error stacktraces as a single group ?

magnusbaeck · August 8, 2018, 12:07pm

That's what a proper multiline configuration should take care of. I don't know what your logs look like so I can't make a concrete suggestion, but the Timestamps example from the documentation is probably what you're looking for.

Raghuveer_SJ · August 9, 2018, 3:58am

The exception stacktrace is a typical JAVA exception. As a sample I can have :

2018-08-03 16:55:06.470 [ThreadPoolTaskScheduler-4] ERROR  o.s.s.s.TaskUtils$LoggingErrorHandler.handleError - Unexpected error occurred in scheduled task.
com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
	at com.mongodb.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:477)
	at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:228)
	at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:96)
	...
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	...
	at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:224)
	... 31 common frames omitted

or

2018-08-07 15:26:02.404 [DefaultQuartzScheduler_Worker-1] ERROR org.slf4j.helpers.SubstituteLogger.error - Unable to start/stop JMX
javax.management.InstanceAlreadyExistsException: com.jolbox.bonecp:type=BoneCP
	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
	...
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

magnusbaeck · August 9, 2018, 7:38am

Okay, so a pretty standard log. Follow the Timestamps example (minor adjustments might be necessary).

Topic		Replies	Views
Error in Logstash conditional expression Logstash	5	1280	March 8, 2018
If grok_parse failure Instead of drop. I want to process everything as greedy data Logstash	4	492	April 27, 2020
Logstash Filter error Logstash	3	1144	February 18, 2018
Aggregate error Logstash	2	636	September 8, 2021
Error : "tags" => [ [0] "_grokparsefailure" ]..Grok filter error Logstash	22	2548	September 18, 2019

GROK filter throws ERROR

Related topics