GeoIP error after logstash 2.3.4 -> 2.4.0 update

Elastic Stack Logstash
1

Hello,

yesterday I updated Logstash from 2.3.4 to 2.4.0 (as well as kibana 4.5 -> 4.6) and ever since I noticed GeoIP data is not added anymore. logstash.log shows the following:

:message=>"Unknown error while looking up GeoIP data", 
:exception=>#<NoMethodError: undefined method `unpack' for nil:NilClass>

I also updated all logstash plugins and currently use logstash-filter-geoip 2.0.7

I tried updating the GeoIP database but the error remains.

This is my filter:

 if [postfix_client_ip] {
        geoip {
            source => "postfix_client_ip"
            database => "/etc/logstash/GeoLite2-City.mmdb"
        }
}

Permissions on the database are "rw r r" with the logstash service user/group as owner.

Does anyone see what I did wrong? I looked at the documentation for logstash 2.4 but could not find any relevant changes with GeoIP.

2

Can you show the full error message, including the stack trace that shows the file and line where the exception was raised?

3

Here is the full line from logstash.log (formatted a little, ip and domain replaced).

{
	:timestamp=>"2016-09-22T12:45:36.191000+0200", 
	:message=>"Unknown error while looking up GeoIP data", 
	:exception=>#<NoMethodError: undefined method `unpack' for nil:NilClass>, 
	:field=>"postfix_client_ip", 
	:event=>#<LogStash::Event:0x77105134 @metadata_accessors=#<LogStash::Util::Accessors:0x27603e41 @store={"type"=>"log", "beat"=>"filebeat"}, @lut={}>, @cancelled=false, @data={"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, @metadata={"type"=>"log", "beat"=>"filebeat"}, @accessors=#<LogStash::Util::Accessors:0xcbf0291 @store={"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, @lut={"tags"=>[{"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, "tags"], "[postfix_client_ip]"=>[{"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, "postfix_client_ip"], "postfix_client_ip"=>[{"message"=>"Sep 22 12:45:25 mail postfix/smtp/smtpd[24331]: connect from somedomain.com[12.345.678.90]", "@version"=>"1", "@timestamp"=>"2016-09-22T10:45:32.018Z", "input_type"=>"log", "source"=>"/var/log/mail/maillog", "offset"=>76280, "type"=>"log", "count"=>1, "fields"=>nil, "beat"=>{"hostname"=>"mail.mydomain.com", "name"=>"mail.mydomain.com"}, "host"=>"mail.mydomain.com", "tags"=>["beats_input_codec_plain_applied", "_grok_postfix_success"], "program"=>"postfix/smtp/smtpd", "postfix_client_hostname"=>"somedomain.com", "postfix_client_ip"=>"12.345.678.90"}, "postfix_client_ip"]}>>, :level=>:error}

No real stacktrace from what I could tell so I added some console output to "/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-2.0.7/lib/logstash/filters/geoip.rb" and it seems the error happens at

def get_geo_data_for_ip(ip)
    ensure_database!
    if (cached = lookup_cache[ip])
      cached
    else
      geo_data = Thread.current[threadkey].send(@geoip_type, ip)
      converted = prepare_geodata_for_cache(geo_data)
      lookup_cache[ip] = converted
      converted
    end
  end

The last console output I get is just before the line
"geo_data = Thread.current[threadkey].send(@geoip_type, ip)"
"threadkey" and "ip" had values set:
threadkey=>"geoip-13290"
ip=>"12.345.678.90"

4

I have the same issue. Let me know if there is any other data I can gather.

5

It seems using the "GeoLite2-City.mmdb" was the problem. I removed the "database" configuration entry for the GeoIP filter so it uses the default which ships with logstash-filter-geoip and everything works again.
Seems GeoLite2 is not working anymore? I don't remember doing anything special to get it to work with 2.3.4

6

I'm racking my brain and I can not set the geoip in version 2.4 or 5.0.beta1 could provide your settings?

7

Sure but it is almost the same as in my op:

# /etc/logstash/conf.d/15-filter-postfix.conf
...
if [postfix_client_ip] {
    geoip {
        source => "postfix_client_ip"
        # database => "/etc/logstash/GeoLite2-City.mmdb"
    }
}
...

I left everything as before and just commented the database line so logstash-filter-geoip uses the default (old format of GeoLite I believe).
Do you have that plugin installed?
<logstash_path>/bin/logstash-plugin list --verbose logstash-filter-geoip

8

/usr/share/logstash/bin/logstash-plugin list --verbose logstash-filter-geoip
logstash-filter-geoip (4.0.3)