Intermittent _rubyexception Errors


(Ken Rogers) #1

Can anyone tell me if the init configuration option is supported and/or necessary in ruby script files, and if so, how to use it?

My script file (yesterday.rb) looks like this:

def register(params)
	@f_timestamp = params['f_timestamp']
	@tsh =         params['tsh']
	@tsm =         params['tsm']
	@tss =         params['tss']
	@ms =          params['ms']
	@file_hour =   params['file_hour']
end

def filter(event)
	date_obj = DateTime.parse(event.get(@f_timestamp))
	tsh = event.get(@tsh)
	tsm = event.get(@tsm)
	tss = event.get(@tss)
	ms = event.get(@ms)
	file_hour = event.get(@file_hour)
	if file_hour == '00' and tsh == '23'
		event.set('f_timestamp',date_obj.year.to_s+' '+date_obj.month.to_s+' '+date_obj.prev_day.day.to_s+' '+tsh+':'+tsm+':'+tss+' '+ms)
	else
		event.set('f_timestamp',date_obj.year.to_s+' '+date_obj.month.to_s+' '+date_obj.day.to_s+' '+tsh+':'+tsm+':'+tss+' '+ms)
	end
	return [event]
end

Here's the filter from my logstash pipeline:

filter {
	grok {
		match => [ "source", "%{GREEDYDATA}/HMS.log_(?<file_year>\d{4})(?<file_month>\d{2})(?<file_day>\d{2})(?<file_hour>\d{2})" ]
	}
	grok {
		match => [
			"message", "^<(?<tsh>\d{2}):(?<tsm>\d{2}):(?<tss>\d{2}).(?<ms>\d{3}) %{NOTSPACE:sev} %{NOTSPACE:subsys} %{NOTSPACE:f1} %{DATA:f2}>%{GREEDYDATA:message}$"
		]
		overwrite => ["message"]
	}

	mutate {
		add_field => {
			"f_timestamp" => "%{file_year}-%{file_month}-%{file_day} %{tsh}:%{tsm}:%{tss}.%{ms}"
		}
	}
	ruby {
		id => "ruby"
		path => "/usr/share/logstash/pipeline/yesterday.rb"
		script_params => {
			"f_timestamp" => "f_timestamp"
			"tsh"         => "tsh"
			"tsm"         => "tsm"
			"tss"         => "tss"
			"ms"          => "ms"
			"file_hour"   => "file_hour"
		}
	}
	date {
		match => ["f_timestamp","yyyy MM dd HH:mm:ss SSS"]
		timezone => "UTC"
	}
	mutate {
		remove_field => [ "t","file_year","file_month","file_day","file_hour","tsh","tsm","tss","ms" ]
	}
}

Here are some sample messages from the log (/opt/servera/HMS.log_20180326022934.log) that's being ingested:

<02:19:07.304 *WRN* BUS 3246:3654 0:0>[handleKeepAliveTimer-dpDaemonMonitor.cpp-674]:Keep Alive Timer for [2, cmgrd] Received, DaemonState=DP_STATE_ACTIVE keepAliveRetryCount=1, keepAliveMaxRetryCount=3
<02:19:09.386 *WRN* CWS 4851:8265 0:0><CalRegMgr-R2-1212:0-294-220-1-0>[sendClearNotify(Cciber.cpp:844)] CWS_MSGTYPE_CLEAR_NOTIFY with ClearNotifyData:mCauseCode:0,mCode:98000
<02:19:10.188 *WRN* LAC 851:865 0:0><CcSpp-R6-1212:0-294-224-1-0>[init(SsReg.cpp:68)] Using default one. OptiCalIC
<02:19:10.136 *WRN* CWR 351:23665 0:0><oCsc-1212:0-274-1161-1-11894>[activatedTimeout(CscSm.cpp:1043)] Clear the buffer, type:BE, delay:0

Running these four messages through logstash (6.2.4) causes intermittent _rubyexceptions and _dateparsefailures. When I examine the f_timestamp of the failed messages they look like this:

? f_timestamp 2018-03-26 02:19:10.188

When the tags are missing the f_timestap looks like this (as expected):

?f_timestamp 2018 3 26 02:19:10 136


(Ken Rogers) #2

The error I get in the logstash log is this:

[2018-06-06T17:59:48,878][ERROR][logstash.filters.ruby ] Could not process event: {:script_path=>"/usr/share/logstash/pipeline/yesterday.rb", :class=>"Java::JavaLang::NullPointerException", :backtrace=>["org.jruby.RubyString.getStringForPattern(RubyString.java:3741)", "org.jruby.RubyString.asRegexpArg(RubyString.java:2405)", "org.jruby.RubyString.subBangNoIter(RubyString.java:2445)", "org.jruby.RubyString.sub_bang(RubyString.java:2398)", "org.jruby.RubyString$INVOKER$i$sub_bang.call(RubyString$INVOKER$i$sub_bang.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrTwoOrNBlock.call(JavaMethod.java:402)", "uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.date.format.RUBY$method$_parse_us$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/date/format.rb:581)", "uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.date.format.RUBY$method$_parse_us$0$__VARARGS__(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/date/format.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.date.format.RUBY$method$_parse$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/date/format.rb:850)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:113)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:198)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "usr.share.logstash.pipeline.yesterday.RUBY$method$filter$0(/usr/share/logstash/pipeline/yesterday.rb:11)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.script.context.RUBY$method$execute_filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby/script/context.rb:55)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.script.context.RUBY$method$execute_filter$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby/script/context.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.script.RUBY$method$execute$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby/script.rb:30)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.script.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby/script.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.RUBY$method$file_script$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby.rb:98)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.RUBY$method$file_script$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby.rb:84)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_ruby_minus_3_dot_1_dot_4.lib.logstash.filters.ruby.RUBY$method$filter$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.4/lib/logstash/filters/ruby.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)",


(Ken Rogers) #3

"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$filter_batch$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:445)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$filter_batch$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$worker_loop$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:424)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:386)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}

The part I can't wrap my head around is sometimes they work, sometimes they fail. I was wondering if it's because I don't have a require 'date' statement in my yesterday.rb file?

Regards

Ken


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.