Summary: I followed the steps outlined in the section "Use a custom endpoint" of the documentation, but I encountered some unexpected behavior. The GEOIP databases are not updating on Elasticsearch ingest nodes. I'm not certain if I made a mistake or if there's an issue with the documentation or GEOIP itself.
Problem Description
Expected Behavior
According to the documentation, I expected the GEOIP databases to update successfully on ingest nodes.
Actual Behavior
However, when checking the status with "GET _ingest/geoip/stats," I see that the updates have not occurred:
{
"stats": {
"successful_downloads": 0,
"failed_downloads": 1,
"total_download_time": 0,
"databases_count": 0,
"skipped_updates": 0,
"expired_databases": 0
},
"nodes": {}
}
Node Configuration
I have a total of 14 Elasticsearch nodes (ELK STACK version: 8.11.1):
- 8 data nodes
- 3 ingest nodes
- 3 master nodes
Troubleshooting Steps
I have taken several steps to troubleshoot the issue:
-
Executed "elasticsearch-geoip" for updating databases, which appeared to complete successfully.
elasticsearch@elasticsearch-5c64694f74-26mxt:~/bin$ ./elasticsearch-geoip -s /geoip/ -t /geoip/ Found GeoIP2-City.mmdb, will compress it to GeoIP2-City.tgz Adding GeoIP2-City.tgz to overview.json overview.json created -
Checked file permissions on the target folder and confirmed they are appropriate.
elasticsearch@elasticsearch-5c64694f74-26mxt:/geoip$ ls -ls total 207891 135949 -rw-r--r--. 1 root root 139210785 Jan 17 16:44 GeoIP2-City.mmdb 71942 -rw-rw-rw-. 1 elasticsearch elasticsearch 73668587 Jan 17 16:48 GeoIP2-City.tgz 1 -rw-rw-rw-. 1 elasticsearch elasticsearch 122 Jan 17 16:48 overview.json -
Made configurations and settings adjustments as described in the documentation, including updating endpoint URLs, restarting Elasticsearch, and confirming settings.
- All nodes have the settings "ingest.geoip.downloader.endpoint: https://mydomain.com/overview.json" in elasticsearch.yml.
- Elasticsearch service on all nodes were restarted after adding the endpoint settings.
GET _cluster/settings?include_defaults "geoip": { "cache_size": "1000", "downloader": { "eager": { "download": "false" }, "enabled": "true", "endpoint": "https://mydomain.com/overview.json" } }- I changed the pull interval from 3 days to 1 day. The cluster settings confirm.
"ingest": { "geoip": { "downloader": { "poll": { "interval": "1d" } } } } -
Tested the connection to the endpoint from all nodes, and all of them can retrieve data from the endpoint.
- Command executed and result from all nodes:
$ curl -XGET https://mydomain.com/overview.json [{ "name": "GeoIP2-City.tgz", "md5_hash": "3fe7b4df652ad2b6679da1f043255fb1", "url": "GeoIP2-City.tgz", "updated": 1705510085337 }] -
Attempted changing the URL part of overview.json to the full URL endpoint, but it did not resolve the issue.
[{ "name": "GeoIP2-City.tgz", "md5_hash": "3fe7b4df652ad2b6679da1f043255fb1", "url": "https://mydomain.com/GeoIP2-City.tgz", "updated": 1705510085337 }] -
Set the log level to trace following the instructions in this article. Observed many log lines, including:
[2024-01-17T12:47:50,093][TRACE][o.e.i.g.DatabaseNodeService] [ingest_1] Not checking databases because geoip databases index does not exist
- Followed instructions to delete the .geoip_databases index from this article, but I cannot see this index in my cluster. I attempted to output all indices with the API call:
GET _cat/indices?format=JSON&bytes=b&expand_wildcards=all
However, there is no .geoip_databases index in the output.
Conclusion
I'm currently stuck and uncertain about the next steps to resolve this issue. I've followed the documentation and conducted troubleshooting steps as outlined, but the problem persists. Any assistance or guidance from the community would be greatly appreciated.
References:
- Documentation: "Manage GEOIP Database Updates"
- Troubleshooting Article: Support - Elastic Knowledge Base
- Deleting the .geoip_databases Index: Support - Elastic Knowledge Base
