Multiline filter - single threaded?

Elastic Stack Logstash
1

Hi All,

it looks like the multi-line filter cannot be used with more than one Logstash worker: I get the following error, when the Logstash gets started with -w N (N > 1):

09:01:17.819 [[main]>worker0] ERROR logstash.pipeline - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {
	"exception"=>#<NoMethodError: undefinedmethod`to_hash_with_metadata'fornil: NilClass>,
	"backtrace"=>["C:/programs/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-multiline-3.0.2/lib/logstash/filters/multiline.rb:273:in `merge'",
	"org/jruby/RubyArray.java:1613:in `each'",
	"org/jruby/RubyEnumerable.java:852:in `inject'",
	"C:/programs/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-multiline-3.0.2/lib/logstash/filters/multiline.rb:272:in `merge'",
	"C:/programs/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-multiline-3.0.2/lib/logstash/filters/multiline.rb:230:in `previous_filter!'",
	"C:/programs/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-multiline-3.0.2/lib/logstash/filters/multiline.rb:173:in `filter'",
	"C:/programs/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'",
	"C:/programs/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'",
	"org/jruby/RubyArray.java:1613:in `each'",
	"C:/programs/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'",
	"C:/programs/logstash/logstash-core/lib/logstash/filter_delegator.rb:41:in `multi_filter'",
	"(eval):303:in `filter_func'",
	"C:/programs/logstash/logstash-core/lib/logstash/pipeline.rb:378:in `filter_batch'",
	"org/jruby/RubyProc.java:281:in `call'",
	"C:/programs/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:201:in `each'",
	"org/jruby/RubyHash.java:1342:in `each'",
	"C:/programs/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:200:in `each'",
	"C:/programs/logstash/logstash-core/lib/logstash/pipeline.rb:377:in `filter_batch'",
	"C:/programs/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'",
	"C:/programs/logstash/logstash-core/lib/logstash/pipeline.rb:331:in `start_workers'"]
}

the multi-line config:

multiline {
	pattern => "^%{TIMESTAMP_ISO8601}\s+"
	negate => true
	what => "previous"
}

Env: Logstash v5.2.2 on windows 10

Has anyone else experienced similar problems?
Thank you!

2

just adding some more details:
the logstash crashes not immediately, but after processing a couple of thousand log entries (~2000 - 12000...)

3

Workaround (or maybe the preferred way?):

move the multi-line processing to codecs, e.g.:

input {
  file {
	path => "C:/log/**/*.log"

	start_position => "beginning"
	sincedb_path => "NUL"
	ignore_older => 0

	codec => multiline {
	  pattern => "^%{TIMESTAMP_ISO8601}\s+"
	  negate => true
	  what => previous
	}		
}
4

There is a multiline filter (deprecated, and the one you initially specified) as well as a multiline codec (the recommended component to use), which you seem to be correctly using in your last example.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.