GeoIP error after logstash 2.3.4 -> 2.4.0 update

Hello,

yesterday I updated Logstash from 2.3.4 to 2.4.0 (as well as kibana 4.5 -> 4.6) and ever since I noticed GeoIP data is not added anymore. logstash.log shows the following:

:message=>"Unknown error while looking up GeoIP data", 
:exception=>#<NoMethodError: undefined method `unpack' for nil:NilClass>

I also updated all logstash plugins and currently use logstash-filter-geoip 2.0.7

I tried updating the GeoIP database but the error remains.

This is my filter:

 if [postfix_client_ip] {
        geoip {
            source => "postfix_client_ip"
            database => "/etc/logstash/GeoLite2-City.mmdb"
        }
}

Permissions on the database are "rw r r" with the logstash service user/group as owner.

Does anyone see what I did wrong? I looked at the documentation for logstash 2.4 but could not find any relevant changes with GeoIP.

Can you show the full error message, including the stack trace that shows the file and line where the exception was raised?

Here is the full line from logstash.log (formatted a little, ip and domain replaced).

{
	:timestamp=>"2016-09-22T12:45:36.191000+0200", 
	:message=>"Unknown error while looking up GeoIP data", 
	:exception=>#<NoMethodError: undefined method `unpack' for nil:NilClass>, 
	:field=>"postfix_client_ip", 
	:event=>#<LogStash::Event:0x77105134 @metadata_accessors=#<LogStash::Util::Accessors:0x27603e41 @store={"type"=>"log", "beat"=>"filebeat"}, @lut={}>, @cancelled=false, @data={"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, @metadata={"type"=>"log", "beat"=>"filebeat"}, @accessors=#<LogStash::Util::Accessors:0xcbf0291 @store={"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, @lut={"tags"=>[{"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, "tags"], "[postfix_client_ip]"=>[{"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, "postfix_client_ip"], "postfix_client_ip"=>[{"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, "postfix_client_ip"]}>>, :level=>:error}

No real stacktrace from what I could tell so I added some console output to "/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-2.0.7/lib/logstash/filters/geoip.rb" and it seems the error happens at

def get_geo_data_for_ip(ip)
    ensure_database!
    if (cached = lookup_cache[ip])
      cached
    else
      geo_data = Thread.current[threadkey].send(@geoip_type, ip)
      converted = prepare_geodata_for_cache(geo_data)
      lookup_cache[ip] = converted
      converted
    end
  end

The last console output I get is just before the line
"geo_data = Thread.current[threadkey].send(@geoip_type, ip)"
"threadkey" and "ip" had values set:
threadkey=>"geoip-13290"
ip=>"12.345.678.90"

I have the same issue. Let me know if there is any other data I can gather.

It seems using the "GeoLite2-City.mmdb" was the problem. I removed the "database" configuration entry for the GeoIP filter so it uses the default which ships with logstash-filter-geoip and everything works again.
Seems GeoLite2 is not working anymore? I don't remember doing anything special to get it to work with 2.3.4

I'm racking my brain and I can not set the geoip in version 2.4 or 5.0.beta1 could provide your settings?

Sure but it is almost the same as in my op:

# /etc/logstash/conf.d/15-filter-postfix.conf
...
if [postfix_client_ip] {
    geoip {
        source => "postfix_client_ip"
        # database => "/etc/logstash/GeoLite2-City.mmdb"
    }
}
...

I left everything as before and just commented the database line so logstash-filter-geoip uses the default (old format of GeoLite I believe).
Do you have that plugin installed?
<logstash_path>/bin/logstash-plugin list --verbose logstash-filter-geoip

/usr/share/logstash/bin/logstash-plugin list --verbose logstash-filter-geoip
logstash-filter-geoip (4.0.3)