GeoIP filter - database file updated - restart required?


(Yao Chim) #1

I have the following filter in Logstash 6.4.2

if [client_ip] !~ /^(10\.|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168|127\.)/ {
    geoip {
        database => "/opt/logstash/GeoLite2.mmdb"
        source => "client_ip"
    }
}

The above filter works fine.

My question - If I download and extract the updated GeoLite2.mmdb every month, do I need to restart Logstash everytime?

I can only find these other related topics:

Thanks in advance.


(Yao Chim) #2

OK I've tested this my self and the answer is yes, you do have to restart Logstash after GeoIP database update.

When I overwrote the GeoIP.mmdb database with an updated version a whole load of errors was dumped out into the log file.

Logstash still processes the events and indexes them into ES but without any GeoIP information in the GeoIP fields.

A restart of Logstash and the GeoIP pipeline works as this obviously reloads the mmdb files.

Log Error example:
[2018-11-06T12:14:46,992][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"a fault occurred in a recent unsafe memory access operation in compiled Java code", "backtrace"=>["com.maxmind.db.Decoder.decodeByType(com/maxmind/db/Decoder.java:166)", "com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:147)", "com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:87)", "com.maxmind.db.Reader.resolveDataPointer(com/maxmind/db/Reader.java:252)", "com.maxmind.db.Reader.get(com/maxmind/db/Reader.java:150)", "com.maxmind.geoip2.DatabaseReader.get(com/maxmind/geoip2/DatabaseReader.java:151)", "com.maxmind.geoip2.DatabaseReader.country(com/maxmind/geoip2/DatabaseReader.java:196)", "org.logstash.filters.GeoIPFilter.retrieveCountryGeoData(org/logstash/filters/GeoIPFilter.java:306)", "org.logstash.filters.GeoIPFilter.handleEvent(org/logstash/filters/GeoIPFilter.java:148)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)", "opt.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_geoip_minus_5_dot_0_dot_3_minus_java.lib.logstash.filters.geoip.invokeOther2:handleEvent(opt/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_filter_minus_geoip_minus_5_dot_0_dot_3_minus_java/lib/logstash/filters//opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/lib/logstash/filters/geoip.rb:111)", "opt.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_geoip_minus_5_dot_0_dot_3_minus_java.lib.logstash.filters.geoip.filter(/opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/lib/logstash/filters/geoip.rb:111)", "opt.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(opt/logstash/logstash_minus_core/lib/logstash/filters//opt/logstash/logstash-core/lib/logstash/filters/base.rb:143)", "opt.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/opt/logstash/logstash-core/lib/logstash/filters/base.rb:143)", "opt.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:do_filter(opt/logstash/logstash_minus_core/lib/logstash/filters//opt/logstash/logstash-core/lib/logstash/filters/base.rb:162)", "opt.logstash.logstash_minus_core.lib.logstash.filters.base.block in multi_filter(/opt/logstash/logstash-core/lib/logstash/filters/base.rb:162)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "opt.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther7:each(opt/logstash/logstash_minus_core/lib/logstash/filters//opt/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "opt.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/opt/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "opt.logstash.logstash_minus_core.lib.logstash.filter_delegator.invokeOther10:multi_filter(opt/logstash/logstash_minus_core/lib/logstash//opt/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)", "opt.logstash.logstash_minus_core.lib.logstash.filter_delegator.multi_filter(/opt/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)", "RUBY.block in initialize((eval):66281)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.block in initialize((eval):66278)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)", "RUBY.block in initialize((eval):66309)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.block in initialize((eval):66306)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)", "RUBY.block in initialize((eval):66353)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.block in initialize((eval):66337)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)", "RUBY.block in filter_func((eval):7130)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "opt.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther3:filter_func(opt/logstash/logstash_minus_core/lib/logstash//opt/logstash/logstash-core/lib/logstash/pipeline.rb:341)", "opt.logstash.logstash_minus_core.lib.logstash.pipeline.filter_batch(/opt/logstash/logstash-core/lib/logstash/pipeline.rb:341)", "RUBY.worker_loop(/opt/logstash/logstash-core/lib/logstash/pipeline.rb:320)", "RUBY.block in start_workers(/opt/logstash/logstash-core/lib/logstash/pipeline.rb:286)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)", "java.lang.Thread.run(java/lang/Thread.java:748)"], :thread=>"#<Thread:0x4c47f548@/opt/logstash/logstash-core/lib/logstash/pipeline.rb:157 sleep>"}


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.