Elasticsearch GeoIP and unknown error

Elastic Stack Elasticsearch
1

Hi,

My elasticsearch DB suddently stopped making geoip resolve on entries, and on the elasticsearch DB logs the following message is logged:

[2022-10-13T15:06:21,875][WARN ][o.e.i.g.GeoIpDownloader ] [host] could not delete old chunks for geoip database [GeoLite2-City.mmdb]

This message is in loop ... don't know why it came ...
I tried googling the error message, no answers ^^'
Does someone as some idea about this ?

UPDATE:
Found another interresting log line saying:
Search rejected due to missing shards [[.geoip_databases][0]]. Consider using allow_partial_search_results setting to bypass this error.

Thanks a lot

2

Hi @ulysse31

What version are you running?

One thing you could try is set this in your elasticsearch.yml and restart elasticsearch (you would need to do it on all the data nodes_

ingest.geoip.downloader.enabled: false

This should try to clean up all thing geoip... (should get rid of the databases and actually clean up the geoip enrich indices

Thne set back to true or take out and restart

ingest.geoip.downloader.enabled: true

That should re-initialize things

See if that helps... could be something else though...

3

If you're getting the "Could Not Resolve Host Elasticsearch Unknown Error" when trying to connect to Elasticsearch, it's likely because your Elasticsearch instance is not accessible from the network. To fix this, you'll need to add the following line to your Elasticsearch.yml file:

network.host: 0.0.0.0

This will bind Elasticsearch to all network interfaces, which will make it accessible from anywhere.

Regards,
Rachel Gomez

4

Hi all,

Disabling and re-enabling again did seem to correct the issue ^^' :

[2022-10-14T09:06:31,512][INFO ][o.e.i.g.DatabaseNodeService] [host] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-10198363484070202212/geoip-databases/s3-uO9IpT1ithOFzkPyVpg/GeoLite2-ASN.mmdb.tmp.gz]
[2022-10-14T09:06:31,523][INFO ][o.e.i.g.GeoIpDownloader ] [host] successfully downloaded geoip database [GeoLite2-ASN.mmdb]
[2022-10-14T09:06:31,606][INFO ][o.e.i.g.DatabaseNodeService] [host] successfully reloaded changed geoip database file [/tmp/elasticsearch-10198363484070202212/geoip-databases/s3-uO9IpT1ithOFzkPyVpg/GeoLite2-ASN.mmdb]
[2022-10-14T09:06:33,615][INFO ][o.e.i.g.DatabaseNodeService] [host] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-10198363484070202212/geoip-databases/s3-uO9IpT1ithOFzkPyVpg/GeoLite2-City.mmdb.tmp.gz]
[2022-10-14T09:06:33,624][INFO ][o.e.i.g.GeoIpDownloader ] [host] successfully downloaded geoip database [GeoLite2-City.mmdb]
[2022-10-14T09:06:34,174][INFO ][o.e.i.g.DatabaseNodeService] [host] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-10198363484070202212/geoip-databases/s3-uO9IpT1ithOFzkPyVpg/GeoLite2-Country.mmdb.tmp.gz]
[2022-10-14T09:06:34,181][INFO ][o.e.i.g.GeoIpDownloader ] [host] successfully downloaded geoip database [GeoLite2-Country.mmdb]
[2022-10-14T09:06:34,236][INFO ][o.e.i.g.DatabaseNodeService] [host] successfully reloaded changed geoip database file [/tmp/elasticsearch-10198363484070202212/geoip-databases/s3-uO9IpT1ithOFzkPyVpg/GeoLite2-Country.mmdb]
[2022-10-14T09:06:34,265][INFO ][o.e.i.g.DatabaseNodeService] [host] successfully reloaded changed geoip database file [/tmp/elasticsearch-10198363484070202212/geoip-databases/s3-uO9IpT1ithOFzkPyVpg/GeoLite2-City.mmdb]

Even if I do not know exactly what caused it :confused: ... Really hope it won't happen again ...
What about the entries that could not resolve and got tagged "_geoip_expired_database" ? will they be automatically corrected ? (I spotted the error in the log because of spotting events being tagged with it initially ^^' )

Thanks a lot again for the help.

Regards,

5

I'm not sure what expired messages you're talking about, but when you disable and re-enable first it loads the default that came with your distribution which is typically out of date and then it will load the latest geoip database.

Elasticsearch will then keep the database up-to-date. I believe it's a 30 day cycle.
.

6

Sorry I might not have been clear enough, let me reformulate and give more informations:

Our elasticsearch DB was updating entries on-the-fly received by filebeat, to insert in the documents geoip informations about the related IPs inside the documents (source/destination IPs).
Since the October 12, I started to see documents with tag "_geoip_expired_database", and without geoip informations on it.
After looking into elasticsearch logs, I spotted the log :

[2022-10-13T15:06:21,875][WARN ][o.e.i.g.GeoIpDownloader ] [host] could not delete old chunks for geoip database [GeoLite2-City.mmdb]

After which I created this discussion.

But, if I give a little overlook to documents inside filebeat indexes created between the 12 and the 13, present documents are still with tag "_geoip_expired_database" and without geoip informations.
I was asking myself, since I just re-enabled GeoIpDownloader, and it seems to work again as it should (log indicating correct download, and new documents getting geoip informations), If inside the geoip feature on the elasticsearch DB, there is a automatic job that would automatically correct the documents with tag "_geoip_expired_database" and set the geoip informations on those documents ?

Thanks a lot

7

Thanks for the explanation... make sense

Unfortunately no .. you would need to re-ingest or perhaps some form of update by query with the ingestion pipeline / geoip processor

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.