Possible logstash fingerprint filter issue?


(David) #1

Hi everyone, apology for the generic title, but I really can't quite find the best title for this thread (feel free to suggest me some)

Quite recently, I checked my Kibana and found out that I have been missing logfile data for 2 days, and when I checked my ELKStask, it turns out that I'm having problem with my Logstash since I'm seeing this message when I restarted my logstash:

[2018-10-01T10:10:18,089][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"64", "backtrace"=>["org.bouncycastle.crypto.digests.SHA256Digest.processWord(Unknown Source)", "org.bouncycastle.crypto.digests.GeneralDigest.update(Unknown Source)", "org.bouncycastle.jcajce.provider.digest.BCMessageDigest.engineUpdate(Unknown Source)", "java.security.MessageDigest.update(Unknown Source)", "org.jruby.ext.openssl.Digest.update(Digest.java:192)", "org.jruby.ext.openssl.Digest$INVOKER$i$1$0$update.call(Digest$INVOKER$i$1$0$update.gen)", "org.jruby.RubyClass.finvoke(RubyClass.java:908)", "org.jruby.runtime.Helpers.invoke(Helpers.java:402)", "org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:363)", "org.jruby.ext.digest.RubyDigest$DigestInstance.digest(RubyDigest.java:319)", "org.jruby.ext.digest.RubyDigest$DigestInstance.hexdigest(RubyDigest.java:339)", "C_3a_.ELKStack.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$method$fingerprint_openssl$0(C:/ELKStack/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:161)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)", "C_3a_.ELKStack.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$block$filter$4(C:/ELKStack/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:140)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "C_3a_.ELKStack.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$method$filter$0(C:/ELKStack/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:135)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb:143)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb:162)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0$__VARARGS__(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(C:/ELKStack/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", ...

Based on the error message, I'm assuming this might have something to do with my logstash fingerprint filter, since I'm seeing "org.bouncycastle.crypto.digests.SHA256Digest.processWord" and I did use SHA256 for my fingerprint plugin after all. However, I couldn't seem to pinpoint the exact issue and doing some googling didn't help me either sadly.

If it helps, here is my pipeline configuration I used for my fingerprint filter.

filter {
	fingerprint{
		source => "message"
		target => "[@metadata][fingerprint]"
		method => "SHA256"
	}
}

I checked the raw log file, and there certainly are data that should've gotten through the filter and be outputted to elasticsearch, so I'm assuming they are stuck at logstash right now.


(David) #2

For some reason, suddenly I'm now getting the missing data between 2 days ago to current time, despite that error message, almost as if logstash finally decided to continue pushing through.

I wonder what that was all about? I guess the whole ELK stack restart helps, despite me still getting that error message after restarting logstash...


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.