GeoIP processor / setup?

Balu · February 24, 2022, 11:21pm

My geoip processor doesn't want to download its data?

ingest.geoip.downloader.enabled is true by default, otherwise I wouldn't be able to query the following stats, right? I have tried enabling it manually as persistent setting too, but that doesn't make a difference.

$ curl -k -X GET "https://localhost:9200/_ingest/geoip/stats?pretty"
{
  "stats" : {
    "successful_downloads" : 0,
    "failed_downloads" : 0,
    "total_download_time" : 0,
    "databases_count" : 0,
    "skipped_updates" : 0,
    "expired_databases" : 0
  },
  "nodes" : { }
}

Do I need to enable something else? To me the stats looks like no attempts were made at all… Also I am confused why it doesn't list any nodes?

I can connect to the Elastic GeoIP endpoint https://geoip.elastic.co/v1/database by using curl, but I only get a 400: Bad Request reply.

stephenb · February 24, 2022, 11:35pm

What Version Stack are you on?

What do your elasticsearch logs look like, usually there is a line on whether it needed to download. Perhaps elasticsearch can not write / download to the temp directory... Look at the logs.

Also are you sure you have the setting correct

From the docs here

Prerequisites

If ingest.geoip.downloader.enabled is disabled, this API returns zero values and an empty nodes object.

Which what your result is.

You should see something like

curl -k -X GET "http://localhost:9200/_ingest/geoip/stats?pretty"
{
  "stats" : {
    "successful_downloads" : 3,
    "failed_downloads" : 0,
    "total_download_time" : 4823,
    "databases_count" : 3,
    "skipped_updates" : 0,
    "expired_databases" : 0
  },
  "nodes" : {
    "6OTEKpyUTTuJ8057Ucd_Tg" : {
      "databases" : [
        {
          "name" : "GeoLite2-ASN.mmdb"
        },
        {
          "name" : "GeoLite2-Country.mmdb"
        },
        {
          "name" : "GeoLite2-City.mmdb"
        }
      ],
      "files_in_temp" : [
        "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb_LICENSE.txt",
        "GeoLite2-City.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb",
        "GeoLite2-City.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb",
        "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-Country.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb",
        "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
        "GeoLite2-Country.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb_README.txt"
      ]
    }
  }
}

Balu · February 25, 2022, 7:33am

I have just upgraded to 8.0.0 on a Debian 10 system.

I am seeing a lot of this request accesses aliases with names reserved for system indices and this request accesses aliases with names reserved for system indices, but no download entries in the current logs.

There is an older entry where it initialized the database registry using a folder in /tmp/elasticsearch* but as far as I can see there isn't a similar folder at this moment.

[2022-02-10T09:12:05,471][INFO ][o.e.i.g.DatabaseNodeService] [elk-elasticsearch-01] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-68890245248798155/geoip-databases/72eUP9qXTgiiQDFb8HmP1g]

I have just restarted one node and the ingest-geoip is loaded, but I don't see any messages on a download.

One thing that confuses me now is that the node is started with -Djava.io.tmpdir=/tmp/elasticsearch-7743237319263154963, but there is no such directory?

I do not have any persistent or transient settings and there is no mention of geoip in elasticsearch.yml, so it should use the default which is true:

$ curl -k -X GET "https://localhost:9200/_cluster/settings?pretty"
{
  "persistent" : { },
  "transient" : { }
}

Balu · February 25, 2022, 10:17am

Ok, I have now done dist-upgrades to Debian 11 on my nodes to be up to date with the whole system and I see a tmp-folder like /tmp/systemd-private-*-elasticsearch.service-*/tmp/elasticsearch-*/geoip-databases/*/ but it is empty.

In the logs I see the geoip module being loaded, but no other information on failed downloads or similar.

# rg -i geoip /var/log/elasticsearch/elk-elasticsearch-prod.log
242:[2022-02-25T09:13:11,971][INFO ][o.e.p.PluginsService     ] [elk-elasticsearch-01] loaded module [ingest-geoip]

Is it possible it is now waiting for the default three days for the next download? If so - is there a way to trigger the download manually?

stephenb · February 25, 2022, 3:03pm

I am confused.. exactly what is your concern... Why do you think you have failed download.

The distribution comes with the latest geoip database so it does not need to download it initially.

It will wait 3 days to check to see if it needs a new download. Not necessarily download a new one every 3 days. MaxMind release is a new database about every 30 days is my understanding

It will / should download a new one when needed.

What is the output now of

curl -k -X GET "https://localhost:9200/_ingest/geoip/stats?pretty"

If you want to force a download you could set

ingest.geoip.downloader.enabled: false
Then restart all the nodes that will clear out the geoip database.

Then set to true and restart that will force a download of the database

Balu · February 25, 2022, 3:30pm

I am confused myself

My concern is missing geoip data for example for the System integration SSH login attemps dashboard. Logs have an source.ip field, but the source.geo.country_iso_code is empty.

The documents also have tags _geoip_database_unavailable_GeoLite2-City.mmdb, _geoip_database_unavailable_GeoLite2-ASN.mmdb set.

curl -k -X GET "https://localhost:9200/_ingest/geoip/stats?pretty"
{
  "stats" : {
    "successful_downloads" : 0,
    "failed_downloads" : 0,
    "total_download_time" : 0,
    "databases_count" : 0,
    "skipped_updates" : 0,
    "expired_databases" : 0
  },
  "nodes" : { }
}

I will try to "turn it off and on again" and report back…

stephenb · February 25, 2022, 4:01pm

Intersesting... OK now I understand your issue.... Yes the DBs appear to be missing and that is why your geoip is not working...

turn off / on and looks at the logs carefully

When you turn back on you should see logs something like (These are from 7.17.0) I have not tested on 8.0 yet

[2022-02-25T08:00:09,535][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] downloading geoip database [GeoLite2-ASN.mmdb] to [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-8287626989138835694/geoip-databases/0gvfvFL5QP6-SJ1MzcQygA/GeoLite2-ASN.mmdb.tmp.gz]
[2022-02-25T08:00:09,549][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updated geoip database [GeoLite2-ASN.mmdb]
[2022-02-25T08:00:09,564][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updating geoip database [GeoLite2-City.mmdb]
[2022-02-25T08:00:09,704][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] successfully reloaded changed geoip database file [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-8287626989138835694/geoip-databases/0gvfvFL5QP6-SJ1MzcQygA/GeoLite2-ASN.mmdb]
[2022-02-25T08:00:13,908][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] downloading geoip database [GeoLite2-City.mmdb] to [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-8287626989138835694/geoip-databases/0gvfvFL5QP6-SJ1MzcQygA/GeoLite2-City.mmdb.tmp.gz]
[2022-02-25T08:00:13,922][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updated geoip database [GeoLite2-City.mmdb]
[2022-02-25T08:00:13,923][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updating geoip database [GeoLite2-Country.mmdb]
[2022-02-25T08:00:15,316][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] downloading geoip database [GeoLite2-Country.mmdb] to [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-8287626989138835694/geoip-databases/0gvfvFL5QP6-SJ1MzcQygA/GeoLite2-Country.mmdb.tmp.gz]
[2022-02-25T08:00:15,331][INFO ][o.e.i.g.GeoIpDownloader  ] [hyperion] updated geoip database [GeoLite2-Country.mmdb]
[2022-02-25T08:00:15,389][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] successfully reloaded changed geoip database file [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-8287626989138835694/geoip-databases/0gvfvFL5QP6-SJ1MzcQygA/GeoLite2-Country.mmdb]
[2022-02-25T08:00:15,705][INFO ][o.e.i.g.DatabaseNodeService] [hyperion] successfully reloaded changed geoip database file [/var/folders/p9/r3gjl2912ybd827msztppr_m0000gn/T/elasticsearch-8287626989138835694/geoip-databases/0gvfvFL5QP6-SJ1MzcQygA/GeoLite2-City.mmdb]

Balu · February 25, 2022, 5:31pm

I guess, the usual "turning it off and on again" fix worked as it always does...

I disabled the geoip processor:

curl -k -X PUT "https://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent" : {
    "ingest.geoip.downloader.enabled" : false
  }
}
'

Then stopped and started each node, waiting for the cluster to recover every time.

Then I set ingest.geoip.downloader.enabled to null (making it default to true) and the system logged:

[2022-02-25T18:24:39,462][INFO ][o.e.c.s.ClusterSettings  ] [node-01] updating [ingest.geoip.downloader.enabled] from [false] to [true]                                                                                         │
[2022-02-25T18:24:41,693][INFO ][o.e.i.g.DatabaseNodeService] [node-01] successfully loaded geoip database file [GeoLite2-ASN.mmdb]                                                                                             │
[2022-02-25T18:24:44,967][INFO ][o.e.i.g.DatabaseNodeService] [node-01] successfully loaded geoip database file [GeoLite2-Country.mmdb]                                                                                         │
[2022-02-25T18:24:45,052][INFO ][o.e.i.g.DatabaseNodeService] [node-01] successfully loaded geoip database file [GeoLite2-City.mmdb]                                                                                            │

Thanks for your help, Stephen

stephenb · February 25, 2022, 6:27pm

Restart / Reboot / Reinstall ... 3 Truths of IT
Glad it is working for you ...

system · March 25, 2022, 6:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.