Hi everyone, apology for the generic title, but I really can't quite find the best title for this thread (feel free to suggest me some)
Quite recently, I checked my Kibana and found out that I have been missing logfile data for 2 days, and when I checked my ELKStask, it turns out that I'm having problem with my Logstash since I'm seeing this message when I restarted my logstash:
[2018-10-01T10:10:18,089][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"64", "backtrace"=>["org.bouncycastle.crypto.digests.SHA256Digest.processWord(Unknown Source)", "org.bouncycastle.crypto.digests.GeneralDigest.update(Unknown Source)", "org.bouncycastle.jcajce.provider.digest.BCMessageDigest.engineUpdate(Unknown Source)", "java.security.MessageDigest.update(Unknown Source)", "org.jruby.ext.openssl.Digest.update(Digest.java:192)", "org.jruby.ext.openssl.Digest$INVOKER$i$1$0$update.call(Digest$INVOKER$i$1$0$update.gen)", "org.jruby.RubyClass.finvoke(RubyClass.java:908)", "org.jruby.runtime.Helpers.invoke(Helpers.java:402)", "org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:363)", "org.jruby.ext.digest.RubyDigest$DigestInstance.digest(RubyDigest.java:319)", "org.jruby.ext.digest.RubyDigest$DigestInstance.hexdigest(RubyDigest.java:339)", "C_3a_.ELKStack.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$method$fingerprint_openssl$0(C:/ELKStack/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:161)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)", "C_3a_.ELKStack.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$block$filter$4(C:/ELKStack/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:140)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "C_3a_.ELKStack.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$method$filter$0(C:/ELKStack/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:135)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb:143)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb:162)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0$__VARARGS__(C:/ELKStack/logstash/logstash-core/lib/logstash/filters/base.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "C_3a_.ELKStack.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(C:/ELKStack/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", ...
Based on the error message, I'm assuming this might have something to do with my logstash fingerprint filter, since I'm seeing "org.bouncycastle.crypto.digests.SHA256Digest.processWord" and I did use SHA256 for my fingerprint plugin after all. However, I couldn't seem to pinpoint the exact issue and doing some googling didn't help me either sadly.
If it helps, here is my pipeline configuration I used for my fingerprint filter.
filter {
fingerprint{
source => "message"
target => "[@metadata][fingerprint]"
method => "SHA256"
}
}
I checked the raw log file, and there certainly are data that should've gotten through the filter and be outputted to elasticsearch, so I'm assuming they are stuck at logstash right now.