# Logstash filter plugin fingerprint failing from unknown reason **URL:** \ **Category:** Logstash\ **Created:** [August 28, 2018, 2:50pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin-fingerprint-failing-from-unknown-reason/146381 "2018-08-28T14:50:21Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [August 28, 2018, 2:50pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin-fingerprint-failing-from-unknown-reason/146381/1 "2018-08-28T14:50:21Z")
Hello, since a few days I am trying to solve a problem of failing logstash. I am downloading ip database using http\_poller. It is approximately 30k IP addresses. I want to change the \_id field using fingerprint filter plugin (it works well). However, it crashes from unknown reason. If I am in normal logstash mode I am able to load approximately 2000 messages, but when I am in full debug mode I am getting like 28k. Please see the configuration below. config file: > input { > http\_poller { > urls =\> { > blocklist\_de\_all =\> "[http://lists.blocklist.de/lists/all.txt](http://lists.blocklist.de/lists/all.txt)" > } > request\_timeout =\> 30 > tags =\> ["blocklist"] > codec =\> "line" > validate\_after\_inactivity =\> 200 > schedule =\> { cron =\> "\*/30 \* \* \* \*" } > metadata\_target =\> "feed\_metadata" > } > } > > filter { > split { > field =\> "[message]" > } > if ([message] =~ /^#/) { > drop{} > } > else { > grok { > match =\> { "message" =\> "^%{GREEDYDATA:ipv4address}" } > } > } > geoip { > source =\> "ipv4address" > add\_tag =\> ["ipv4enriched"] > add\_field =\> ["[geoip][coordinates]", "%{[geoip][longitude]}" ] > add\_field =\> ["[geoip][coordinates]", "%{[geoip][latitude]}" ] > } > mutate { > convert =\> ["[geoip][coordinates]", "float"] > } > > ``` > fingerprint { > id => "blocklist1" > source => ["ipv4address"] > method => ["SHA512"] > add_tag => ["fingerprinted"] > > ``` > > } > } > > output { > elasticsearch { > hosts =\> ["10.0.50.51:9200"] > index =\> "ipv4\_to\_block" > document\_id =\> "%{fingerprint}" > document\_type =\> "default" > } > } pipeline config: > - pipeline.id: blocklist\_ips > path.config: "/etc/logstash/conf.d/blocklist\_de\_all\_low\_confidence.conf" > pipeline.workers: 16 I am getting what I want in Elasticsearch (output from Kibana): > { > "\_index": "ipv4\_to\_block", > "\_type": "default", > **"\_id": "1eda4277c8b054652a08b0f56f26656babbe8328",** > "\_version": 1, > "\_score": 1, > "\_source": { > **"fingerprint": "1eda4277c8b054652a08b0f56f26656babbe8328",** > "@version": "1", > "metadata": { > "host": "elk2", > "name": "blocklist\_de\_all", > "request": { > "method": "get", > "url": "[http://lists.blocklist.de/lists/all.txt](http://lists.blocklist.de/lists/all.txt)" > }, > "code": 200, > "response\_message": "OK", > "runtime\_seconds": 0.115168, > "times\_retried": 0, > "response\_headers": { > "connection": "keep-alive", > "content-type": "text/plain; charset=UTF-8", > "transfer-encoding": "chunked", > "date": "Tue, 28 Aug 2018 14:16:55 GMT", > "last-modified": "Tue, 28 Aug 2018 14:14:10 GMT", > "cache-control": "public", > "x-frame-options": "sameorigin", > "keep-alive": "timeout=20", > "server": "nginx/1.12.2", > "etag": "W/"6550b-5747f74a5da2f"" > } > }, > "tags": [ > "blocklist", > "\_geoip\_lookup\_failure", > "fingerprinted" > ], > "ipv4address": "103.115.180.188", > "@timestamp": "2018-08-28T14:21:00.411Z", > "message": "103.115.180.188", > "geoip": {} > }, > "fields": { > "@timestamp": [ > "2018-08-28T14:21:00.411Z" > ] > } > } Configured logstash in debug mode: > curl -XPUT 'localhost:9600/\_node/logging?pretty' -H 'Content-Type: application/json' -d' > { > "logger.logstash.agent" : "DEBUG", > "logger.logstash.api.service" : "DEBUG", > "logger.logstash.codecs.json" : "DEBUG", > "logger.logstash.codecs.line" : "DEBUG", > "logger.logstash.codecs.plain" : "DEBUG", > "logger.logstash.config.source.local.configpathloader" : "DEBUG", > "logger.logstash.config.source.multilocal" : "DEBUG", > "logger.logstash.config.sourceloader" : "DEBUG", > "logger.logstash.configmanagement.extension" : "DEBUG", > "logger.logstash.filters.drop" : "DEBUG", > "logger.logstash.filters.grok" : "DEBUG", > "logger.logstash.filters.split" : "DEBUG", > "logger.logstash.inputs.http\_poller" : "DEBUG", > "logger.logstash.instrument.periodicpoller.deadletterqueue" : "DEBUG", > "logger.logstash.instrument.periodicpoller.jvm" : "INFO", > "logger.logstash.instrument.periodicpoller.os" : "DEBUG", > "logger.logstash.instrument.periodicpoller.persistentqueue" : "DEBUG", > "logger.logstash.modules.scaffold" : "DEBUG", > "logger.logstash.modules.xpackscaffold" : "DEBUG", > "logger.logstash.monitoringextension" : "DEBUG", > "logger.logstash.monitoringextension.pipelineregisterhook" : "DEBUG", > "logger.logstash.outputs.elasticsearch" : "DEBUG", > "logger.logstash.outputs.file" : "DEBUG", > "logger.logstash.pipeline" : "INFO", > "logger.logstash.plugins.registry" : "DEBUG", > "logger.logstash.runner" : "DEBUG", > "logger.org.logstash.Logstash" : "DEBUG", > "logger.org.logstash.common.DeadLetterQueueFactory" : "DEBUG", > "logger.org.logstash.common.io.DeadLetterQueueWriter" : "DEBUG", > "logger.org.logstash.config.ir.CompiledPipeline" : "DEBUG", > "logger.org.logstash.instrument.metrics.gauge.LazyDelegatingGauge" : "DEBUG", > "logger.org.logstash.plugins.pipeline.PipelineBus" : "DEBUG", > "logger.org.logstash.secret.store.SecretStoreFactory" : "DEBUG", > "logger.slowlog.logstash.codecs.json" : "DEBUG", > "logger.slowlog.logstash.aodecs.line" : "DEBUG", > "logger.slowlog.logstash.codecs.plain" : "DEBUG", > "logger.slowlog.logstash.filters.drop" : "DEBUG", > "logger.slowlog.logstash.filters.grok" : "DEBUG", > "logger.slowlog.logstash.filters.split" : "DEBUG", > "logger.slowlog.logstash.inputs.http\_poller" : "DEBUG", > "logger.slowlog.logstash.outputs.elasticsearch" : "DEBUG", > "logger.slowlog.logstash.outputs.file" : "DEBUG" > } > ' Why the logstash loads less data if not being in full debug mode? Why this configuration fails anyway? Is it because it cannot handle the load of hashing 30k or failing on the local ip database lookup? Thank you for the help in advance. ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [August 28, 2018, 2:51pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin-fingerprint-failing-from-unknown-reason/146381/2 "2018-08-28T14:51:14Z")
The error I get is: > [2018-08-28T16:24:01,488][ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline\_id=\>"blocklist\_ips", "exception"=\>"8", "backtrace"=\>["org.bouncycastle.crypto.digests.LongDigest.update(Unknown Source)", "org.bouncycastle.crypto.digests.LongDigest.finish(Unknown Source)", "org.bouncycastle.crypto.digests.SHA512Digest.doFinal(Unknown Source)", "org.bouncycastle.jcajce.provider.digest.BCMessageDigest.engineDigest(Unknown Source)", "java.security.MessageDigest.digest(MessageDigest.java:365)", "org.jruby.ext.openssl.Digest.finish(Digest.java:204)", "org.jruby.ext.openssl.Digest$INVOKER$i$0$0$finish.call(Digest$INVOKER$i$0$0$finish.gen)", "org.jruby.RubyClass.finvoke(RubyClass.java:557)", "org.jruby.runtime.Helpers.invoke(Helpers.java:399)", "org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:354)", "org.jruby.ext.digest.RubyDigest$DigestInstance.digest(RubyDigest.java:320)", "org.jruby.ext.digest.RubyDigest$DigestInstance.hexdigest(RubyDigest.java:339)", "org.jruby.ext.digest.RubyDigest$DigestInstance$INVOKER$s$0$1$hexdigest.call(RubyDigest$DigestInstance$INVOKER$s$0$1$hexdigest.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:721)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET\_METHOD(MixedModeIRMethod.java:179)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_3\_dot\_0.gems.logstash\_minus\_filter\_minus\_fingerprint\_minus\_3\_dot\_2\_dot\_0.lib.logstash.filters.fingerprint.RUBY$block$filter$4(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:140)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "usr.share.logstash.vendor.bundle.jruby.$2\_dot\_3\_dot\_0.gems.logstash\_minus\_filter\_minus\_fingerprint\_minus\_3\_dot\_2\_dot\_0.lib.logstash.filters.fingerprint.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:135)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.filters.base.RUBY$method$do\_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.filters.base.RUBY$block$multi\_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.filters.base.RUBY$method$multi\_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.filter\_delegator.RUBY$method$multi\_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter\_delegator.rb:44)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET\_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.pipeline.RUBY$method$filter\_batch$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:340)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.pipeline.RUBY$method$worker\_loop$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:319)", "usr.share.logstash.logstash\_minus\_core.lib.logstash.pipeline.RUBY$block$start\_workers$2(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:285)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"], :thread=\>"#\"} > [2018-08-28T16:24:01,490][ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline\_id=\>"blocklist\_ips", "exception"=\>"8", "backtrace"=\>["org.bouncycastle.crypto.digests.LongDigest.update(Unknown Source)", "org.bouncycastle.crypto.digests.LongDigest.update(Unknown Source)", "org.bouncycastle.jcajce.provider.digest.BCMessageDigest.engineUpdate(Unknown Source)", "java.security.MessageDigest.update(MessageDigest.java:325)", "org.jruby.ext.openssl.Digest.update(Digest.java:192)", "org.jruby.ext.openssl.Digest$INVOKER$i$1$0$update.call(Digest$INVOKER$i$1$0$update.gen)", > ... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 25, 2018, 2:51pm UTC](https://discuss.elastic.co/t/logstash-filter-plugin-fingerprint-failing-from-unknown-reason/146381/3 "2018-09-25T14:51:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.