Nginx-ingress-controller-access, _geoip_database_unavailable_GeoLite2-City.mmdb, _geoip_database_unavailable_GeoLite2-ASN.mmdb

[Subrahmanyam_Veerank (Subramanyam v)]

Hi sir,
Enabled the nginx ingress controller integration on elastic agent (kubernetes env) but the tags related to geo locations which were populated earlier are now not coming. kindly guide on this part.

tags
[nginx-ingress-controller-access, _geoip_database_unavailable_GeoLite2-City.mmdb, _geoip_database_unavailable_GeoLite2-ASN.mmdb]

Tried the below commands but the issue still persists.

PUT _cluster/settings
{
"transient": {
"ingest.geoip.downloader.enabled" : false
}
}

PUT _cluster/settings
{
"transient": {
"ingest.geoip.downloader.enabled" : true
}
}

as per the previous topic: Elastic Observability Engineer Lab 5.4 - geoip database unavailable

[Subrahmanyam_Veerank (Subramanyam v)]

Is there any solution for this issue?

[Subrahmanyam_Veerank (Subramanyam v)]

GET _ingest/geoip/stats

{
  "stats": {
    "successful_downloads": 0,
    "failed_downloads": 1,
    "total_download_time": 0,
    "databases_count": 0,
    "skipped_updates": 0,
    "expired_databases": 0
  },
  "nodes": {}
}

[stephenb (Stephen Brown)]

Hi @Subrahmanyam_Veerank

It looks like the databases are failing to download.

The GEOIP Databases are downloaded from the Internet.

See here

GeoIP processor | Elasticsearch Guide [8.12] | Elastic

Can you reach this from your elasticsearch server

https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agre

[Subrahmanyam_Veerank (Subramanyam v)]

Hi sir..

It is reachable when the proxy is enabled on the server. Actually we kept the proxy disabled in all the nodes.

ELK02:~$ curl -I https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree
HTTP/1.1 200 Connection established

HTTP/2 200
date: Tue, 27 Feb 2024 04:37:42 GMT
content-type: application/json
content-length: 2996
vary: Accept-Encoding
permissions-policy: interest-cohort=()
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
strict-transport-security: max-age=31536000; includeSubDomains

i have enabled the proxy on all the nodes but still the geoip database download is getting failed.
kindly guide us to proceed further

[stephenb (Stephen Brown)]

@Subrahmanyam_Veerank

i have enabled the proxy on all the nodes but still the geoip database download is getting failed.
kindly guide us to proceed further

Can your try this? and report back

PUT _cluster/settings
{
  "transient": {
    "ingest.geoip.downloader.eager.download": true
  }
}

Then run this again

GET _ingest/geoip/stats

Also if you look at the details of

curl https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree

You will notice the actual downloads come from google

So your proxy would need to support that as well.

https://storage.googleapis.com/elastic-paisano-production/maxmind/GeoLite2-ASN.tgz

[Subrahmanyam_Veerank (Subramanyam v)]

Sir..

Tried below cluster setting.

PUT _cluster/settings
{
  "transient": {
    "ingest.geoip.downloader.eager.download": true
  }
}

  "stats": {
    "successful_downloads": 0,
    "failed_downloads": 2,
    "total_download_time": 0,
    "databases_count": 0,
    "skipped_updates": 0,
    "expired_databases": 0
  },
  "nodes": {}
}

ELKM01:~$ curl -I https://storage.googleapis.com/elastic-paisano-production/maxmind/GeoLite2-ASN.tgz
HTTP/1.1 200 Connection established

HTTP/2 403
x-guploader-uploadid: ABPtcPrrfGpxlYj4cK2BRmvIC3VmvH-_OldTtqrm797LoGPZnRhtk9IL-fkWqa3rtHxPwe2CVfk
content-type: application/xml; charset=UTF-8
content-length: 298
date: Wed, 28 Feb 2024 05:57:46 GMT
expires: Wed, 28 Feb 2024 05:57:46 GMT
cache-control: private, max-age=0
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

it is able to connect but still it is unable to download the dbs.

[stephenb (Stephen Brown)]

Hmmmm .....

The good part it tried again. The bad part it failed.

So now we need to go a bit deeper and this is going to get a little more complex especially because you have 10 nodes.

There are most likely logs in the elasticsearch logs that will have further information.

Also what I do sometimes in these tough things is I just set up a single node to debug on so I don't have the complexity of 10 nodes.

Possible causes are something still not connecting right? Or it loads the files into a /tmp That does not have write permissions.

So look at the logs. See if there's something obvious.

We might need to turn up the debug for the GEO IP component and then look at the logs afterwards.

If I get a chance I'll try to simulate something on my side and see if I can give you some commands.

But check the logs.

I don't know what kind of nodes you have, but it turns out it's only downloaded on data nodes. So if you have coordinator only or ingest only, it will not be downloaded on those.

Oh when you do any testing now do the three commands

Set. Download false
Set. Download to true
Then set the eager

[Subrahmanyam_Veerank (Subramanyam v)]

sir..we have 3 master nodes and 3 (data & ingest) nodes

Set. Download false
Set. Download to true
Then set the eager

yes sir did the same.

[stephenb (Stephen Brown)]

@Subrahmanyam_Veerank

sir..we have 3 master nodes and 3 (data & ingest) nodes

Haha where the heck did I get 10 node from? (again, helping too many people )

So go look in the logs on those data nodes see if you see anything about the GEOIP if you do post it back here... I got a busy day. Might be able to look at it later

[Subrahmanyam_Veerank (Subramanyam v)]

ok sir thank you

[Subrahmanyam_Veerank (Subramanyam v)]

[2024-02-28T11:30:54,760][INFO ][o.e.c.s.ClusterSettings  ] [node2] updating [ingest.geoip.downloader.enabled] from [false] to [true]
[2024-02-28T11:31:05,015][ERROR][o.e.i.g.GeoIpDownloader  ] [node2] exception during geoip databases update
java.net.SocketTimeoutException: Connect timed out
        at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
        at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.net.Socket.connect(Socket.java:751) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
        at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1237) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1675) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[?:?]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:157) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:140) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:161) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:60) ~[?:?]
        at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

[Subrahmanyam_Veerank (Subramanyam v)]

[2024-02-28T11:26:35,584][ERROR][o.e.i.g.GeoIpDownloader  ] [node3] exception during geoip databases update
java.net.SocketTimeoutException: Connect timed out
        at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
        at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.net.Socket.connect(Socket.java:751) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
        at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1237) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1675) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[?:?]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:157) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:140) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) ~[elasticsearch-8.11.4.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-02-28T11:

[stephenb (Stephen Brown)]

Can you try to curls from that node which it's failing?

The error is pretty self-explanatory cannot connect...

Are you using a proxy? That is most likely the issue....

I think there might be a proxy setting. Go look at that page for the GEOIP

From the doc's

In a strict setup the following domains may need to be added to the allowed domains list:

geoip.elastic.co
storage.googleapis.com

[Subrahmanyam_Veerank (Subramanyam v)]

yes sir we were using the proxy but it is able to connect.

ELK03:~$ curl -I https://storage.googleapis.com/elastic-paisano-production/maxmind/GeoLite2-ASN.tgz
HTTP/1.1 200 Connection established

HTTP/2 403
x-guploader-uploadid: ABPtcPoOnZx666e8c7KzE2oXUYFo3A1JoDhCG_RRu4ZGGH6NbJim96UqjDGkuTUlVi7GN2UkGUM
content-type: application/xml; charset=UTF-8
content-length: 298
date: Wed, 28 Feb 2024 06:17:42 GMT
expires: Wed, 28 Feb 2024 06:17:42 GMT
cache-control: private, max-age=0
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

[stephenb (Stephen Brown)]

There is a connectivity issue

You are going to need to turn up the logs to trace for that component...

Any way you can try without the Proxy?

[Subrahmanyam_Veerank (Subramanyam v)]

Add support to configure Proxy to allow geoip databases update

we were facing the same issue here.

[stephenb (Stephen Brown)]

Interesting did you see the workaround halfway down

Add support to configure Proxy to allow geoip databases update

I successfully use a proxy without authentication to update geoip database with ES_JAVA_OPTS with this 2 kinds of method :

• "ES_JAVA_OPTS=-Djava.net.useSystemProxies=true"
  (With the proxy system configured through https_proxy environment configured)

or

• "ES_JAVA_OPTS=-Dhttps.proxyHost=10.x.x.x -Dhttps.proxyPort=3128 -Dhttps.nonProxyHosts=localhost|127.0.0.1|10.*.*.*"

So the ES_JAVA_OPTS seems to work now.

There are other methods manual download if needed...

[Subrahmanyam_Veerank (Subramanyam v)]

yes sir tried. but it didnt work.

[stephenb (Stephen Brown)]

What did you try?

No proxy?

Did you try the JVM settings?

Not sure what to tell you there is a connectivity issue of some sort.

Turn on the trace logs and look for further information

You may need to manually download