How to disable geoip usage in 7.14.0

Elastic Stack Elasticsearch
1

hi, I want to disable the usage of geoip functionality because of licensing and no usage . I have a single node cluster, which works fine. I've tried to disable the geoip on a cluster level using the following param: ingest.geoip.downloader.enabled=false. But I'm getting exceptions after ES restart : trying to update state on non-existing task geoip-downloader
[2021-08-11T15:33:57,318][ERROR][o.e.i.g.GeoIpDownloader ] [18789989a729] error updating geoip database [GeoLite2-Country.mmdb]
Is there any way to disable completely the geoip usage?

thanks in advance

2

Same here.. Found the following cluster setting.. tried setting it in elasticsearch.yml but it's still trying to do the GeoIPDownloader update.

ingest.geoip.downloader.enabled: false
1 Like
3

">

5

same issue when deploying es7.14.0 by Helm. this is the elasticsearch.yaml in the container.

[elasticsearch@elasticsearch-master-0 config]$ cat elasticsearch.yml 
http.cors.enabled: true
http.cors.allow-origin: "*"
indices.breaker.total.limit: 80%
ingest.geoip.downloader.enabled: false
6

Hi @Alex_Batt and @jasonpan Welcome to the community, apologies you are having issues.

Thanks @AndrewMcQuerry Thanks for confirming...

I pinged internally, this is a brand new feature as I think you know. The engineers that worked on this change are both OOO at this time.

One question to help isolate / reproduce

Did you all start your 7.14 the first time with this setting on first start?

ingest.geoip.downloader.enabled=false

Or did you

  1. start the node / cluster without the setting
  2. Then stop the node / cluster
  3. Set the setting
  4. Then Start The node / cluster

My understanding is that should still work but I'm just curious on reproducing the steps.

@Alex_Batt Are you saying you can enable / disable fine on a Single Node Cluster but not on a MultiNode Cluster?

I am going to try to reproduce as well...

7

I only set the setting after seeing the message in the logs, then searching to find out what it could be about, then finding the setting and trying it.

I have not yet tried starting a fresh 7.14 with that parameter already set.

8

Same for me after Update from 7.13.4. Saw the Log entry after the update start... found

ingest.geoip.downloader.enabled: false

but with no luck the "false" will be ignored

9

I was able to reproduce this and can confirm that it's real issue. I've opened #76586 to track it and I'll work on a fix.

For now a workaround should be to use cluster settings API instead of elasticsearch.yml:

PUT _cluster/settings
{
    "persistent": {
       "ingest.geoip.downloader.enabled": true
    }
}

PUT _cluster/settings
{
    "persistent": {
       "ingest.geoip.downloader.enabled": false
    }
}

(it has to be enabled first for cluster to notice change, no need for restart between and after these calls)

4 Likes
10

Hi @Przemko_Robakowski, can you describe step by step how to implement the “cluster settings API” solution. Or do you have a link to a solution?

11

You need to put that into Kibana Devtools Console.

12

Hi @Przemko_Robakowski , thank you. I’ll try the suggested workaround. Is there a planned date that a fix will be delivered as part of an official version?

13

Hi, in my Case, to test the API Workaround. I create a complete new test elk-stack. And after that i will do a elasticsearch restart to test it and saw the behavior in the Logs.

:~# systemctl stop elasticsearch.service
:~# systemctl start elasticsearch.service

and saw in the Log

:~# tail -f /var/log/elasticsearch/elasticsearch.log
[...]
[2021-08-17T14:48:30,422][INFO ][o.e.i.g.LocalDatabases   ] [devubu2004] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2021-08-17T14:48:30,444][INFO ][o.e.i.g.DatabaseRegistry ] [devubu2004] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-808219805137404262/geoip-databases/YRNSL6wtTTC5J6Ili4f7nw]
[...]
[2021-08-17T14:48:39,689][INFO ][o.e.c.s.ClusterSettings  ] [devubu2004] updating [ingest.geoip.downloader.enabled] from [true] to [false]
[...]
[2021-08-17T14:48:44,419][INFO ][o.e.i.g.GeoIpDownloader  ] [devubu2004] updating geoip databases
[2021-08-17T14:48:44,437][INFO ][o.e.i.g.GeoIpDownloader  ] [devubu2004] fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]
[2021-08-17T14:48:46,911][INFO ][o.e.i.g.GeoIpDownloader  ] [devubu2004] updating geoip database [GeoLite2-ASN.mmdb]
[2021-08-17T14:48:46,939][INFO ][o.e.c.r.a.AllocationService] [devubu2004] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.geoip_databases][0]]]).
[2021-08-17T14:48:52,085][WARN ][o.e.p.PersistentTasksClusterService] [devubu2004] trying to update state on non-existing task geoip-downloader
[2021-08-17T14:48:52,091][ERROR][o.e.i.g.GeoIpDownloader  ] [devubu2004] error updating geoip database [GeoLite2-ASN.mmdb]
org.elasticsearch.ResourceNotFoundException: the task with id geoip-downloader and allocation id 2 doesn't exist
        at org.elasticsearch.persistent.PersistentTasksClusterService$4.execute(PersistentTasksClusterService.java:236) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:691) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:313) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:208) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:62) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:140) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204) ~[elasticsearch-7.14.0.jar:7.14.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

[... three times ERROR...]

Is/should that ERROR now the normal behavior?

And after that i want to delete the new created und mybe useless Index

green open .geoip_databases EDd8AnYfS8-8epV7Opua_g 1 0 84 0 81.3mb 81.3mb

:~$ curl -XDELETE 'http://localhost:9200/.geoip_databases'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Indices [.geoip_databases] use and access is reserved for system operations"}],"type":"illegal_argument_exception","reason":"Indices [.geoip_databases] use and access is reserved for system operations"},"status":400}

But i can't do that. Also is this the normal behavior?

14

Hi @localhorst

If you create a new elasticsearch AND set the setting below in the elasticsearch.yml BEFORE you ever start the node / cluster it will not try to download the geoip database. I have tested / verified this

ingest.geoip.downloader.enabled: false

It still loads the local default geoip database that is pre-license change as I understand it.

It will not try to download the new geoip database

1 Like
15

Hi @stephenb,

this is good to know for future new installations. But i have some test and produktion elk-stack updated from 7.13.4 and now to figure out without to create everything new.

1 Like
16

hi I've tested the suggested workaround, as @localhorst mentioned I'm also getting an error:
[2021-08-22T15:01:39,961][WARN ][o.e.p.PersistentTasksClusterService] [966549702ac3] trying to update state on non-existing task geoip-downloader
[2021-08-22T15:01:39,962][ERROR][o.e.i.g.GeoIpDownloader ] [966549702ac3] error updating geoip database [GeoLite2-Country.mmdb]
org.elasticsearch.ResourceNotFoundException: the task with id geoip-downloader and allocation id 10 doesn't exist
at org.elasticsearch.persistent.PersistentTasksClusterService$4.execute(PersistentTasksClusterService.java:236) ~[elasticsearch-7.14.0.jar:7.14.0]

Is this the expected behavior?

17

after restart i still can see the following in the logs:

[2021-08-22T15:40:04,052][INFO ][o.e.i.g.LocalDatabases ] [b6099ef08b0b] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes
[2021-08-22T15:40:04,054][INFO ][o.e.i.g.DatabaseRegistry ] [b6099ef08b0b] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-12593652174417475614/geoip-databases/lhXMxtDyTuGnlnHJWOqUMg]

though I don't see the geoip extra index created.
is this as expected?

18

Did you see the post above...
There is a bug That has been identified

Also What It would help if you could tell us what steps you took to get to this.

Was it a fresh install?

Was it an upgrade?

At which point did you try setting the setting It helps us understand the issue

19

Tested 2 scenarios:

  1. I've installed and run the ES, when it was up and running I've applied the suggested workaround. (with the put commands) and got the exception in the logs:
    restart : trying to update state on non-existing task geoip-downloader
    [2021-08-11T15:33:57,318][ERROR][o.e.i.g.GeoIpDownloader ] [18789989a729] error updating geoip database [GeoLite2-Country.mmdb]

  2. I used clean installation and set ingest.geoip.downloader.enabled: false before starting the ES.
    in this case i saw in logs:
    [2021-08-22T15:40:04,052][INFO ][o.e.i.g.LocalDatabases ] [b6099ef08b0b] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes
    [2021-08-22T15:40:04,054][INFO ][o.e.i.g.DatabaseRegistry ] [b6099ef08b0b] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-12593652174417475614/geoip-databases/lhXMxtDyTuGnlnHJWOqUMg]

but didn't see an attempt to update the DB.

@stephenb, are both behaviors are as expected?

20

Scenario 2 : Yes that looks correct, only the default Geo DB is initialized and no new Geo DB is attempted to be downloaded.

Scenario 1 : I got the same error, I believe that is related to the bug filed above

@Przemko_Robakowski

Can you confirm the following error in the elasticsearch startup logs, this is after we applied the settings you referred to above and then restarted the cluster is related to the bug you filed?

[2021-08-23T09:58:41,776][INFO ][o.e.i.g.GeoIpDownloader  ] [ceres] updating geoip database [GeoLite2-City.mmdb]
[2021-08-23T09:58:45,526][WARN ][o.e.p.PersistentTasksClusterService] [ceres] trying to update state on non-existing task geoip-downloader
[2021-08-23T09:58:45,528][ERROR][o.e.i.g.GeoIpDownloader  ] [ceres] error updating geoip database [GeoLite2-City.mmdb]
org.elasticsearch.ResourceNotFoundException: the task with id geoip-downloader and allocation id 2 doesn't exist
        at org.elasticsearch.persistent.PersistentTasksClusterService$4.execute(PersistentTasksClusterService.java:236) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48) ~[elasticsearch-7.14.0.jar:7.14.0]
21

Thank you very much for the report, I thought it was the only one.

This issue is fixed with versions after 7.14.0, I mean 7.14.1 or 7.15?