Disabling geoip downloader is not working (elastic 7.16.2 and java 11)

Elastic Stack Elasticsearch
1

RHEL release 7.9
jdk-11.0.14 (not bundled)
Elasticsearch with x-pack security multi hosts/nodes cluster has been setup from Elasticsearch-7.16.2-linux-x86_64.tar.gz. It is fully functional.

I tried to disable geoip downloader in Elasticsearch.yml on each and every node of the cluster with this line

ingest.geoip.downloader.enabled: false

which is not working and I can see appropriate subfolder geoip-databases in the elastic logs subdirectory.
Is there any other config file, options I can use to disable geoip ?

Any help on this issue will be appreciated

Thanks in advance

2

Hi @dzyubanv

Hmm ... That is the correct setting. There was a bug back in 7. 13 but that was fixed in 7.14 unless there's been some regression.

Did you originally start the cluster with the download on / default setting?

Can you show the startup logs?

If you run

GET _cat/indices/.geo*/?v

Do you see
green open .geoip_databases

I just ran 7.16.3 and it seems to be working (on a single node... are you sure all nodes are set?... just asking)

3

I can't share the log of the cluster and communicate on the forum from corp machine due to the security policy.

I setup single node cluster to test this geoip feature using exactly the same environment and configuration as I used in multi-node cluster.
On this setup i place the same line in Elasticsearch.yml

ingest.geoip.downloader.enabled: false

and after starting node I can see in logs the geoip-databases folder.
The log file contains the following lines:

.............................  
Plugin Service loaded module [ingest geoip]
ConfigDatabases   initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb]]
initialized database registry, using geoip-databases directory
.............................

In the above I truncated some lines, characters for simplicity typing.
Node started gracefully, the health of cluster is green.

Running GET _cat/indices/.geo*/?v gave me one line in the response:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

Also, I found the same issue with geoip has been reported on Aug 16, 2021 on elastic version 7.14 at GeoIpDownloader can't be disable by elasticsearch.yml · Issue #76586 · elastic/elasticsearch · GitHub by Przemko Robakowski

Thank you for the response Stephen and if you can provide other guidelines or recommendations I would gladly to follow them.
Many thanks for the help and attention.

4

Ok... Looking Close...

Ok I downloaded a Fresh 7.16.2 and only added 1 line

ingest.geoip.downloader.enabled: false

You will still see the initialization of the "default" database upon startup BUT there is no actual geoip database loaded into elasticsearch... which you already actually confirmed...(see below)

You will see these logs lines.. which I think are a bit misleading....

[2022-04-16T19:45:41,128][INFO ][o.e.i.g.ConfigDatabases  ] [hyperion] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/Users/sbrown/workspace/elastic-install/7.16.2/tmp/elasticsearch-7.16.2/config/ingest-geoip] for changes
[2022-04-16T19:45:41,130][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] initialized database registry, using geoip-databases directory [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-6246939774672860089/geoip-databases/C7v7WTzGQ7CnHKoZ8sbmAw]

If you then run....

hyperion:elasticsearch-7.16.2 sbrown$ curl http://127.0.0.1:9200/_cat/indices/.geo*?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

That means there is no .geo ip indexes / databases are actually loaded into elasticsearch ... i.e. they are not loaded INTO elasticsearch.

This also confirms it... the geoip stats show no downloads.

hyperion:elasticsearch-7.16.2 sbrown$ curl localhost:9200/_ingest/geoip/stats
{"stats":{"successful_downloads":0,"failed_downloads":0,"total_download_time":0,"databases_count":0,"skipped_updates":0,"expired_databases":0},"nodes":{}}hyperion:elasticsearch-7.16.2

Now lets enabled it and start again...
ingest.geoip.downloader.enabled: true

You will see logs like this.....notr the updated and downloaded databases

[2022-04-16T19:50:11,468][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] successfully reloaded changed geoip database file [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-15885661857466451462/geoip-databases/C7v7WTzGQ7CnHKoZ8sbmAw/GeoLite2-ASN.mmdb]
[2022-04-16T19:50:15,055][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] downloading geoip database [GeoLite2-City.mmdb] to [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-15885661857466451462/geoip-databases/C7v7WTzGQ7CnHKoZ8sbmAw/GeoLite2-City.mmdb.tmp.gz]
[2022-04-16T19:50:15,080][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updated geoip database [GeoLite2-City.mmdb]
[2022-04-16T19:50:15,081][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updating geoip database [GeoLite2-Country.mmdb]
[2022-04-16T19:50:16,373][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] downloading geoip database [GeoLite2-Country.mmdb] to [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-15885661857466451462/geoip-databases/C7v7WTzGQ7CnHKoZ8sbmAw/GeoLite2-Country.mmdb.tmp.gz]
[2022-04-16T19:50:16,387][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updated geoip database [GeoLite2-Country.mmdb]
[2022-04-16T19:50:16,443][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] successfully reloaded changed geoip database file [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-15885661857466451462/geoip-databases/C7v7WTzGQ7CnHKoZ8sbmAw/GeoLite2-Country.mmdb]
[2022-04-16T19:50:16,711][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] successfully reloaded changed geoip database file [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-15885661857466451462/geoip-databases/C7v7WTzGQ7CnHKoZ8sbmAw/GeoLite2-City.mmdb]

and now we can see the geoip index etc and the stats about the download updates...

hyperion:elasticsearch-7.16.2 sbrown$ curl http://127.0.0.1:9200/_cat/indices/.geo*?v
health status index            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .geoip_databases IJM4aKLmTB2YlxEfWfwkMA   1   0         40            0     37.9mb         37.9mb

hyperion:elasticsearch-7.16.2 sbrown$ curl localhost:9200/_ingest/geoip/stats
{"stats":{"successful_downloads":3,"failed_downloads":0,"total_download_time":7733,"databases_count":3,"skipped_updates":0,"expired_databases":0},"nodes":{"C7v7WTzGQ7CnHKoZ8sbmAw":{"databases":[{"name":"GeoLite2-City.mmdb"},{"name":"GeoLite2-Country.mmdb"},{"name":"GeoLite2-ASN.mmdb"}],"files_in_temp":["GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt","GeoLite2-ASN.mmdb_LICENSE.txt","GeoLite2-City.mmdb_LICENSE.txt","GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt","GeoLite2-ASN.mmdb","GeoLite2-City.mmdb_COPYRIGHT.txt","GeoLite2-City.mmdb","GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt","GeoLite2-Country.mmdb_LICENSE.txt","GeoLite2-Country.mmdb","GeoLite2-ASN.mmdb_COPYRIGHT.txt","GeoLite2-Country.mmdb_COPYRIGHT.txt","GeoLite2-City.mmdb_README.txt"]}}}hyperion:elasticsearch-7.16.2

Now disable again
ingest.geoip.downloader.enabled: false

now it went back to the initialized the default database (the one that comes shipped) BUT does not actually load the databases into elasticsearch.

[2022-04-16T19:52:56,306][INFO ][o.e.i.g.ConfigDatabases  ] [hyperion] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/Users/sbrown/workspace/elastic-install/7.16.2/tmp/elasticsearch-7.16.2/config/ingest-geoip] for changes
[2022-04-16T19:52:56,307][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] initialized database registry, using geoip-databases directory [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-1698940183761466503/geoip-databases/C7v7WTzGQ7CnHKoZ8sbmAw]
....
[2022-04-16T19:52:59,909][INFO ][o.e.c.m.MetadataDeleteIndexService] [hyperion] [.geoip_databases/IJM4aKLmTB2YlxEfWfwkMA] deleting index

And now there is no .geoip index (i.e. it was cleaned up)

hyperion:elasticsearch-7.16.2 sbrown$ curl http://127.0.0.1:9200/_cat/indices/.geo*?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

and it not longer recognizes the downloaded databases

hyperion:elasticsearch-7.16.2 sbrown$ curl localhost:9200/_ingest/geoip/stats
{"stats":{"successful_downloads":0,"failed_downloads":0,"total_download_time":0,"databases_count":0,"skipped_updates":0,"expired_databases":0},"nodes":{}}hyperion:elasticsearch-7.16.2

Now I do still see the "Downloaded" temp database but it is not longer used...

I would need to get clarification of what the docs mean but after you set to false no new updates will come, and the geoip databases are not actually loaded into elasticsearch.

If false , Elasticsearch does not download updates and deletes all downloaded databases.

So I think there are 2 things ... the message about initializing the default (shipped) database is a bit confusing even though it is not Actually loaded into Elasticsearch

and the fact the the temp downloads are not cleaned up.. perhaps they should be... but when set to false the geoip database are not loaded into .. nor updated into elasticsearch.

5

BTW you can use the following and observe the same behavior so technically you do not need to put in each Elasticsearch.yml. If you use persistent the setting will persist over restarts.

PUT /_cluster/settings
{
  "persistent": { 
  "ingest.geoip.downloader.enabled" : "false"
  }
}


PUT /_cluster/settings
{
  "persistent": { 
  "ingest.geoip.downloader.enabled" : "true"
  }
}

PUT /_cluster/settings
{
  "persistent": { 
  "ingest.geoip.downloader.enabled" : "false"
  }
}
6

Thank you very much Stephen for the comprehensive investigation and explanation of this elastic geoip setup and behavior and sharing your experience globally.
Definitely it might be helpful for other elastic users too.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.