6.7 docker crashes on ingest-geoip - expected database [GeoLite2-ASN.mmdb] to not exist in [/usr/share/elasticsearch/config/ingest-geoip]

mortenb123 · March 27, 2019, 12:43pm

I'm unable to get geoip-ingest to work with latest 6.7 elastic search container

When I start:
docker.elastic.co/elasticsearch/elasticsearch 6.7.0 02982be5777d 5 days ago 810MB

[2019-03-27T10:28:55,197][INFO ][o.e.p.PluginsService     ] [5O5yY3c] loaded module [ingest-common]
[2019-03-27T10:28:55,198][INFO ][o.e.p.PluginsService     ] [5O5yY3c] loaded module [ingest-geoip]
[2019-03-27T10:28:55,198][INFO ][o.e.p.PluginsService     ] [5O5yY3c] loaded module [ingest-user-agent]
:
[2019-03-27T10:28:58,925][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [5O5yY3c] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/usr/share/elasticsearch/config/ingest-geoip]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.7.0.jar:6.7.0]

/usr/share/elasticsearch/config is an external volume: mounted on /mnt/data/esconfig:
$ ls -l /mnt/data/esconfig/ingest-geoip/
total 69564
-rw-rw---- 1 sensio root 6455484 Mar 27 11:17 GeoLite2-ASN.mmdb
-rw-rw---- 1 sensio root 61034576 Mar 27 11:29 GeoLite2-City.mmdb
-rw-rw---- 1 sensio root 3733305 Mar 27 11:28 GeoLite2-Country.mmdb

I even downloaded the most recent ones from maxmind.

If I try just without the esconfig volume, ingest geoip dir is not created:

The directory /usr/share/elasticsearch/config/ does not have an ingest-geoip:
-rw-rw---- 1 elasticsearch root 207 Mar 27 11:44 elasticsearch.keystore
-rw-r--r-- 1 elasticsearch root 53 Mar 21 15:34 elasticsearch.yml
-rw-rw---- 1 elasticsearch root 3613 Mar 21 15:29 jvm.options
-rw-r--r-- 1 elasticsearch root 285 Mar 21 15:34 log4j2.properties
-rw-rw---- 1 elasticsearch root 473 Mar 21 15:34 role_mapping.yml
-rw-rw---- 1 elasticsearch root 197 Mar 21 15:34 roles.yml
-rw-rw---- 1 elasticsearch root 0 Mar 21 15:34 users
-rw-rw---- 1 elasticsearch root 0 Mar 21 15:34 users_roles

The config does not say anything about where the geo-ip config directory resides:
https://www.elastic.co/guide/en/elasticsearch/reference/master/geoip-processor.html

mortenb123 · March 27, 2019, 12:48pm

it is located in the plugin, but how do it set this to point here:
/usr/share/elasticsearch/modules/ingest-geoip
[root@90f234ed7cce ingest-geoip]# ls -l
total 67196
-rw-r--r-- 1 elasticsearch root 6173457 Mar 21 15:34 GeoLite2-ASN.mmdb
-rw-r--r-- 1 elasticsearch root 57784030 Mar 21 15:34 GeoLite2-City.mmdb
-rw-r--r-- 1 elasticsearch root 3428908 Mar 21 15:34 GeoLite2-Country.mmdb
-rw-r--r-- 1 elasticsearch root 44701 Mar 21 15:34 geoip2-2.9.0.jar
-rw-r--r-- 1 elasticsearch root 26467 Mar 21 15:34 ingest-geoip-6.7.0.jar
-rw-r--r-- 1 elasticsearch root 55786 Mar 21 15:34 jackson-annotations-2.8.11.jar
-rw-r--r-- 1 elasticsearch root 1247857 Mar 21 15:34 jackson-databind-2.8.11.3.jar
-rw-r--r-- 1 elasticsearch root 21693 Mar 21 15:34 maxmind-db-1.2.2.jar
-rw-r--r-- 1 elasticsearch root 1511 Mar 21 15:34 plugin-descriptor.properties
-rw-r--r-- 1 elasticsearch root 1461 Mar 21 15:34 plugin-security.policy

Janko · March 27, 2019, 1:32pm

Hi @mortenb123, sorry to hear you are having troubles.

I have just spun up a freshly pulled image and it loads OK:

[2019-03-27T13:13:43,246][INFO ][o.e.p.PluginsService     ] [es0] loaded module [ingest-common]
[2019-03-27T13:13:43,246][INFO ][o.e.p.PluginsService     ] [es0] loaded module [ingest-geoip]
[2019-03-27T13:13:43,246][INFO ][o.e.p.PluginsService     ] [es0] loaded module [ingest-user-agent]
[2019-03-27T13:13:43,247][INFO ][o.e.p.PluginsService     ] [es0] loaded module [lang-expression]

It seems the one you have running from 5 days ago and we released the final only Yesterday. Could you make sure that you have pulled the latest version?
Do you make any other changes, and could you paste your exact commands used to try and reproduce this?

Thanks!

mukai · March 27, 2019, 10:31pm

This problem may be caused by the following changes.

Elasticsearch 6.7.0 checks that there are no leftover geoip database files in the plugin configuration directory because the new module does not use them. Therefore, remove the ingest-geoip plugin prior to upgrading to 6.7.0 with the --purge option to also delete the old database files:
> ./bin/elasticsearch-plugin remove --purge ingest-geoip
Otherwise you will see the following error message upon startup (assuming /etc/elasticsearch/ingest-geoip as the plugin configuration directory):
expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.7.html#_literal_ingest_geoip_literal_and_literal_ingest_user_agent_literal_are_no_longer_distributed_as_plugins

mortenb123 · March 28, 2019, 10:53am

you still get the same elasticsearch tag on:

sensio@logtastic:/dist/docker-containers/elastic-enterprise$ docker rmi  docker.elastic.co/elasticsearch/elasticsearch:6.7.0
Untagged: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
Untagged: docker.elastic.co/elasticsearch/elasticsearch@sha256:58ae94b778f880cf90433404a3524324feeafedebf5c85f6639a9c0f2f2b070b
Deleted: sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38
Deleted: sha256:4d7fa31d0f53239e73ebc237921a0a46106c2b7e05378a9ea5370a0519e9f0d4
Deleted: sha256:8f2e5618d0d852252eec04b67cf56a0bc215bd01ea56daaf6253f8c5e6c4471c
Deleted: sha256:4aaca09d61801844c24e7e44357cb224f3ecba95cd5a5a44dc8999ed659a9a63
Deleted: sha256:31ee0c61de7820cfa2c3bb7908bdc860e3926e7879f2580027eed3f85b62bb97
Deleted: sha256:d7b6bfabd5f9ffe622b48880e5c11a859ef091dceb69b5f9d8acb4326de8753f
Deleted: sha256:9a2222f159b68964211dca263481014fc8be05f2d6cd9104b2e1976d724f5dbc
sensio@logtastic:/dist/docker-containers/elastic-enterprise$ docker pull  docker.elastic.co/elasticsearch/elasticsearch:6.7.0
6.7.0: Pulling from elasticsearch/elasticsearch
8ba884070f61: Already exists
d841587f1077: Pull complete
a1e10c27c0c9: Pull complete
c2f79ed21c46: Pull complete
b3709b240809: Pull complete
7f8bbbc43a34: Pull complete
eb30f8913021: Pull complete
Digest: sha256:58ae94b778f880cf90433404a3524324feeafedebf5c85f6639a9c0f2f2b070b
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:6.7.0
sensio@logtastic:/dist/docker-containers/elastic-enterprise$ docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
docker.elastic.co/kibana/kibana                 6.7.0               a19f604cd838        6 days ago          675MB
docker.elastic.co/elasticsearch/elasticsearch   6.7.0               02982be5777d        6 days ago          810MB

purging the plugin fixed it, thanks

system · April 25, 2019, 10:53am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.