Logstash 5.0.0-alpha4 + geoip = Something is wrong with your configuration

Elastic Stack Logstash
1

Hi

I'm experiencing some issues with logstash 5.0.0-alpha4 and geoip plugin. Posting it here before opening any issue in case is my fault.

Get to the point.
Run logstash container with GeoIP volume:

docker run --rm -it --hostname 'geoip' --memory 512m --name geoip
--volume /etc/localtime:/etc/localtime:ro 
--volume /var/logstash/config/:/etc/logstash/
--volume /var/logstash/geoip/conf.d/:/etc/logstash/conf.d/
--volume /usr/share/GeoIP/:/usr/share/GeoIP/
 logstash:5.0.0-alpha4 logstash -f /etc/logstash/conf.d/ --config.reload.automatic --config.reload.interval 10

Output:

--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Invalid setting for geoip filter plugin:

  filter {
    geoip {
      # This setting must be a path
      # File does not exist or cannot be opened /usr/share/GeoIP/GeoLiteCity.dat
      database => ["/usr/share/GeoIP/GeoLiteCity.dat"]
      ...
    }
  } {:level=>:error}
  fetched an invalid config {:config=>"input {\n\tredis...<snip>...\"ready\"\n\t}\n}\n\n",:reason=>"Something is wrong with your configuration.", :level=>:error}

Dear Logstash: would you mind to be more specific?

conf.d file (Other paths are used for memory/logging configuration):

input {
    redis {
        host => "redis"
        data_type => "list"
        key => "app"
    }
}
filter {
    if [ip] {
        geoip {
            database => ["/usr/share/GeoIP/GeoLiteCity.dat"]
            source => "ip"
        }
    }
}
output {
    redis {
        host => "redis"
        data_type => "list"
        key => "ready"
    }
}

Double checked file permissions inside container:

ls -la /usr/share/GeoIP/
total 8
drwxr-xr-x  2 root root 4096 Jul 11 10:57 .
drwxr-xr-x 72 root root 4096 Jul 20 15:45 ..
lrwxrwxrwx  1 root root   41 Jan 13  2016 GeoIP.dat -> /var/lib/geoip-database-contrib/GeoIP.dat
lrwxrwxrwx  1 root root   46 Jan 13  2016 GeoIPASNum.dat -> /var/lib/geoip-database-contrib/GeoIPASNum.dat
lrwxrwxrwx  1 root root   48 Jan 13  2016 GeoIPASNumv6.dat -> /var/lib/geoip-database-contrib/GeoIPASNumv6.dat
lrwxrwxrwx  1 root root   31 Jul 11 10:57 GeoIPCity.dat -> /etc/alternatives/GeoIPCity.dat
lrwxrwxrwx  1 root root   43 Jan 13  2016 GeoIPv6.dat -> /var/lib/geoip-database-contrib/GeoIPv6.dat
lrwxrwxrwx  1 root root   47 Jan 13  2016 GeoLiteCity.dat -> /var/lib/geoip-database-contrib/GeoLiteCity.dat
lrwxrwxrwx  1 root root   49 Jan 13  2016 GeoLiteCityv6.dat -> /var/lib/geoip-database-contrib/GeoLiteCityv6.dat

Same conf.d is working with logstash 2.3.2-1.

Any clue? Shall I open the issue @ github?

Thanks!

2

Hi @mostolog,

I am able to reproduce this issue exactly as you have it. It seems that the required format for Geo Databases in 5.x in .mmdb format. I have filed this issue to track documentation.

In the meantime, you should be able to omit the database setting and default to the shipped .mmdb file, or download the desired DB in the appropriate format.

3

We figured that out, as Logstash 2.3 is using GeoLite legacy DB format while logstash 5 uses GeoLite2 DB,
Thanks :wink:

4

Apart from the file format issue, the files in /usr/share/GeoIP are symlinks to files in /var/lib/geoip-database-contrib. Are those file available to the Docker container?

5

Sorry for delay...I was enjoying my holidays!

We are just mounting /usr/share/GeoIP, if that is what you are asking.