_geoip_database_unavailable_GeoLite2-ASN.mmdb

Akjal · April 25, 2023, 9:21pm

I have opened my firewall to receive all traffic on all ports (ipv4 and piv6). but it is still not updating. I am not sure what led to it failing before because it was working for over 3 months with no issue and I didnt need to install geo ip manually. I just followed this doc to set it up Enrich events with geoIP information | Packetbeat Reference [8.7] | Elastic @stephenb

stephenb · April 25, 2023, 9:28pm

It looks like you are actually using 8.4.3 not 8.7?

I can not tell you why it stopped... I do know that Elastic comes with a default GEO DB which expires... and then it will try to download the new one.

The download is failing the per

GET _ingest/geoip/stats

Did you actually try to curl the endpoint?

$ curl https://geoip.elastic.co/v1/database
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>400 Bad Request</title>
</head>
<body align="center">
    <div role="main" align="center">
        <h1>400: Bad Request</h1>
        <p>The request could not be understood by the server due to malformed syntax.</p>
        <hr />
    </div>
    <div role="contentinfo" align="center">
        <small>Rocket</small>
    </div>
</body>

Akjal · April 25, 2023, 9:35pm

sorry for not being clear. I am running 8.4.3 and I ran curl https://geoip.elastic.co/v1/database on my elk node
and this is the output @stephenb

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>400 Bad Request</title>
</head>
<body align="center">
    <div role="main" align="center">
        <h1>400: Bad Request</h1>
        <p>The request could not be understood by the server due to malformed syntax.</p>
        <hr />
    </div>
    <div role="contentinfo" align="center">
        <small>Rocket</small>
    </div>
</body>```

stephenb · April 25, 2023, 9:45pm

Ok so that is good...

The actual endpoint to test (just found it)

curl https://geoip.elastic.co/v1/database/?elastic_geoip_service_tos=agree

Let me take a look I think I just reproduced... but I have some other things now... let me look at get back.

{
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "ip": "89.160.20.128",
    "tags": [
      "_geoip_database_unavailable_GeoLite2-City.mmdb"
    ]
  }
}

Akjal · April 25, 2023, 10:03pm

Ok just inputted the curl comand. I am getting:

[{"age":6899,"md5_hash":"a2ab548cdc96da24b12603f906acb707","name":"GeoLite2-ASN.tgz","provider":"maxmind","updated":1682380830,"url":"https://storage.googleapis.com/elastic-paisano-production/maxmind/GeoLite2-ASN.tgz?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=elastic-paisano-production%40elastic-apps-163815.iam.gserviceaccount.com%2F20230425%2Fhenk%2Fstorage%2Fgoog4_request&X-Goog-Date=20230425T000030Z&X-Goog-Expires=86400&X-Goog-SignedHeaders=host&X-Goog-Signature=57cfb5fc701c645a6c27acbe9081d6f5db990cbde18b87af906d2a32bbe033665b312d27a375fff748d9761f1b9a1509a8e8ded4d5eeceb5fdf45c801f850c92b7dad392d504404a5147f2ac4e8bf6e82fea5e538e77573b9dc983a9b60638f994f84597f5b2e442f3b740c24d398da3832c8dc56b4fc7553d6754b5ed1caead32436caa9b48b20bdb6aba2e60daf6cb5a80f6dfb7b99d998ac00298cae4f76f43651bfa8c44402d59e86c3685b9a0e806ce6d043289e810e0187d4776677a7330095ebb8f5cae51482ee4bea84882e062a35bb0b2c606dbca2a6d8844f2886e06c56ad976f161ff59d8ea1f0b5efd74b5619034ad03cebce04edd877fee3113"},{"age":6822,"md5_hash":"f2bf63338e5e86f6d2f1bd75458fcf63","name":"GeoLite2-City.tgz","provider":"maxmind","updated":1682380831,"url":"https://storage.googleapis.com/elastic-paisano-production/maxmind/GeoLite2-City.tgz?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=elastic-paisano-production%40elastic-apps-163815.iam.gserviceaccount.com%2F20230425%2Fhenk%2Fstorage%2Fgoog4_request&X-Goog-Date=20230425T000031Z&X-Goog-Expires=86400&X-Goog-SignedHeaders=host&X-Goog-Signature=710a5e71e32f3f21d1eb76054dd01ea650a482d1c587b17d31044701b19900321f402bfa5d4b8e2f303a53078a973f872d9e52cdda81f661d78f6e537f880c418d58280f1baf12c6f15acccbbaeb9a6aafe08d4b05fcce47d47387fcf15231d3c865833072528e8f123931be51644f0c591ff8190c8cf23baf5b218c0187fb2fb4b032a79d23560b90dcb0ec785f43c3a96370d6fea6e1eb52970ba5ce95add3e464827e994b348f224cb8b9cb9aa519012f84dad86c4a547669864281eb5847914d5ba0ec5b0916c4cea04ecfab54093dafeeff67be416a390394117f9e303e21f5cec9212e2a4650673766b3422b5a3838ca8e3ff00799434418f87b0a7707"},{"age":6817,"md5_hash":"572bf983015458f1fb91408b2a9515ba","name":"GeoLite2-Country.tgz","provider":"maxmind","updated":1682380830,"url":"https://storage.googleapis.com/elastic-paisano-production/maxmind/GeoLite2-Country.tgz?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=elastic-paisano-production%40elastic-apps-163815.iam.gserviceaccount.com%2F20230425%2Fhenk%2Fstorage%2Fgoog4_request&X-Goog-Date=20230425T000030Z&X-Goog-Expires=86400&X-Goog-SignedHeaders=host&X-Goog-Signature=036482c8302e28c77f22ab9b6148e37ed3014b6208420c5e40077ff90046ba9bfabc67a7c10a77dd96912a75e7c9c08e714e8fd3dc34ab642232ae339a863965c93e27a88d5d60b9f3f0923c3f5710a927fc8e0576d1026e3d82cc2fbe8bf2a8509a0559e1f7524e804cd6d3e19f009551daa7577a6a7b8c70590907c062399a0c08d0c152a2d82ec3e46445326553ed245f5b103b4d2843f831d9df303657be334e4af9e44224759d1f36072c6be5833873387c41c5f3286966618f77dc14b60936046ddb8fde4b497c34420d32fe656e9ae3a9960a7b2c03b4dbbc480bc2fa81e66599fa238b8ffa5e79df37ed5b13eb812b66e8608d1e137130b6ad48ff31"}]

@stephenb

stephenb · April 25, 2023, 11:33pm

Ok Looks good!

Can you share your elasticsearch.yml

Can you stop and start the elastic node and watch for the events with geoip ... you should see some log entries and share them

also run

GET _cat/indices/.ge*?v

GET _cat/indices/.ge*?v
green  open   .geoip_databases V3wf3F-_QdS5v9PgNRhjmg   1   0         42            0     40.5mb         40.5mb

Also go to
cd /tmp

and look for an elasticsearch directory and go down into it and you should find the databases... that is where the downloaded puts them... then they get loaded into a hidden index

OK TEST

Its weird I got the same result an then I was checking around and it went away..... hmmm

Brand New OOTB Setup 8.7.0

I run
GET _cat/indices/.ge*?v

Blank

I can see in my elasticsearch logs geo database is not loaded..

Then I run

PUT _ingest/pipeline/geoip
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "ip"
      }
    }
  ]
}

PUT my-index-000001/_doc/my_id?pipeline=geoip
{
  "ip": "89.160.20.128"
}


GET my-index-000001/_doc/my_id

# Bad results

{
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "ip": "89.160.20.128",
    "tags": [
      "_geoip_database_unavailable_GeoLite2-City.mmdb"
    ]
  }
}

NOW look in the logs and I see the geo database loaded

I run

GET _cat/indices/.ge*?v
health status index            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .geoip_databases V3wf3F-_QdS5v9PgNRhjmg   1   0         42            0     40.5mb         40.5mb

The geo enrich index...

So now I run ... and I get a good result.

PUT my-index-000001/_doc/my_id?pipeline=geoip
{
  "ip": "89.160.20.128"
}

GET my-index-000001/_doc/my_id

# Good Result
{
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 2,
  "_seq_no": 1,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "ip": "89.160.20.128",
    "geoip": {
      "continent_name": "Europe",
      "region_iso_code": "SE-AB",
      "city_name": "Sollentuna",
      "country_iso_code": "SE",
      "country_name": "Sweden",
      "region_name": "Stockholm County",
      "location": {
        "lon": 17.9653,
        "lat": 59.4284
      }
    }
  }
}

Try my test and see what happens...

Not Sure... seems like there is a delay loading the geo database OR is needs to be primed once (perhaps a bug) I will try another test.

Lets make sure the simple test works

Run the GET
Then my Simple Test
Then Run the Get
Then Run the Test again

Report Back

Akjal · April 26, 2023, 12:09am

here is my elasticsearch.yml file:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 23-11-2022 17:28:14
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ip-172-31-3-15.ec2.internal"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

stephenb · April 26, 2023, 12:13am

Ok elasticsearch.yml looks good for what I was looking for .. run my tests and let me know.

and Make sure /tmp is writable

Akjal · April 26, 2023, 12:14am

I inputed the three command on dev tools:


GET _cat/indices/.ge*?v


PUT my-index-000001/_doc/my_id?pipeline=geoip
{
  "ip": "89.160.20.128"
}



GET my-index-000001/_doc/my_id

This is the output for the last command:

{
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 2,
  "_seq_no": 1,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "ip": "89.160.20.128",
    "tags": [
      "_geoip_database_unavailable_GeoLite2-Country.mmdb"
    ]
  }
}

Plus I went to /tmp and found this folders:

elasticsearch-15494635544508648  elasticsearch-608083247790626  hsperfdata_ec2-user  systemd-private-c6f94cc8479c46a6b2c1670574e-chronyd.service-IRvqTX

stephenb · April 26, 2023, 12:21am

run the commands again

Akjal · April 26, 2023, 12:25am

still getting

{
  "_index": "my-index-000001",
  "_id": "my_id",
  "_version": 3,
  "_seq_no": 2,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "ip": "89.160.20.128",
    "tags": [
      "_geoip_database_unavailable_GeoLite2-Country.mmdb"
    ]
  }
}```

stephenb · April 26, 2023, 12:26am

stop the node clear the logs and start the node and share all the logs with geo in the content

Go down into the latest /tmp after is starts and see if you see the geo databases they should be in there

stephenb · April 26, 2023, 12:42am

Understood

I really did not want you to run restart

I wanted you to stop
Clear out the logs
Start

Then Show the logs that have geo in them.

I need to see the logs that pertain to the geodatabase on startup. They will be there...

I'm very familiar with starting with systemctl.

Also, did you search down into the /tmp and see that the geo databases were actually there.

I am asking these questions for a reason. I need to see all the answers I'm asking... We are now in the territory of something unusual happening and I am asking specific questions so I can understand.

I need to see the logs
I need to see that the database files are in the latest? /tmp/elasticsearch... Directory

Are you logged in as the elastic user?

Akjal · April 26, 2023, 12:59am

where do i fnd the logs,like which logs.

plus I appreciate your help through this really
@stephenb

stephenb · April 26, 2023, 1:21am

Per the docs

/var/log/elasticsearch

should be the clustername.log so in your case my-application.log

Also show results of...

cd /tmp
find . | grep -i geo

Akjal · April 26, 2023, 2:35am

the output for the find . | grep -i geo is :

./elasticsearch-15494635544361508648/geoip-databases
./elasticsearch-15494635544361508648/geoip-databases/QZfXbrEpSretLKsQA3Yrnw
./elasticsearch-6080834973247790626/geoip-databases
./elasticsearch-6080834973247790626/geoip-databases/QZfXbrEpSretLKsQA3Yrnw
./elasticsearch-18018805462673851213/geoip-databases
./elasticsearch-18018805462673851213/geoip-databases/QZfXbrEpSretLKsQA3Yrnw
./elasticsearch-723601772435016518/geoip-databases
./elasticsearch-723601772435016518/geoip-databases/QZfXbrEpSretLKsQA3Yrnw

the output for my-application.log is as follows. I deleted the log file first, stopped elasticsearch and then started back again and got the following logs. not sure if they are helpful :

[2023-04-26T02:01:16,166][INFO ][o.e.n.Node               ] [node-1] version[8.4.3], pid[14866], build[tar/42f05b9372a9a4a470db3b52817899b99a76ee73/2022-10-04T07:17:24.662462378Z], OS[Linux/5.10.147-133.644.amzn2.x86_64/amd64], JVM[Oracle Corporation/Op$
[2023-04-26T02:01:16,168][INFO ][o.e.n.Node               ] [node-1] JVM home [/home/ec2-user/elasticsearch-8.4.3/jdk], using bundled JDK [true]
[2023-04-26T02:01:16,169][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile$
[2023-04-26T02:01:18,847][INFO ][c.a.c.i.j.JacksonVersion ] [node-1] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubles$
[2023-04-26T02:01:20,405][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [aggs-matrix-stats]
[2023-04-26T02:01:20,410][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [analysis-common]
[2023-04-26T02:01:20,410][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [constant-keyword]
[2023-04-26T02:01:20,410][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [data-streams]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [frozen-indices]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-attachment]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-common]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-geoip]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-user-agent]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [kibana]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-expression]
[2023-04-26T02:01:20,411][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-mustache]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-painless]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [legacy-geo]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-extras]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-version]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [old-lucene-versions]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [parent-join]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [percolator]
[2023-04-26T02:01:20,412][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [rank-eval]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [reindex]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repositories-metering-api]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-azure]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-encrypted]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-gcs]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-s3]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-url]
[2023-04-26T02:01:20,413][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [runtime-fields-common]
[2023-04-26T02:01:20,414][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [search-business-rules]
[2023-04-26T02:01:20,414][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [searchable-snapshots]
[2023-04-26T02:01:20,414][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [snapshot-based-recoveries]
[2023-04-26T02:01:20,417][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [snapshot-repo-test-kit]
[2023-04-26T02:01:20,417][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [spatial]
[2023-04-26T02:01:20,417][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transform]
[2023-04-26T02:01:20,417][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transport-netty4]
[2023-04-26T02:01:20,418][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [unsigned-long]
[2023-04-26T02:01:20,418][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [vector-tile]
[2023-04-26T02:01:20,418][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [wildcard]
[2023-04-26T02:01:20,418][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-aggregate-metric]
[2023-04-26T02:01:20,418][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-analytics]
[2023-04-26T02:01:20,418][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-async]
[2023-04-26T02:01:20,419][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-async-search]
[2023-04-26T02:01:20,419][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-autoscaling]
[2023-04-26T02:01:20,419][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ccr]
[2023-04-26T02:01:20,419][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-core]
[2023-04-26T02:01:20,419][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-deprecation]
[2023-04-26T02:01:20,420][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-enrich]
[2023-04-26T02:01:20,420][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-eql]
[2023-04-26T02:01:20,420][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-fleet]
[2023-04-26T02:01:20,420][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-graph]
[2023-04-26T02:01:20,420][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-identity-provider]
[2023-04-26T02:01:20,420][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ilm]
[2023-04-26T02:01:20,420][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-logstash]
[2023-04-26T02:01:20,421][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ml]
[2023-04-26T02:01:20,421][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-monitoring]
[2023-04-26T02:01:20,421][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ql]
[2023-04-26T02:01:20,421][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-rollup]
[2023-04-26T02:01:20,421][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2023-04-26T02:01:20,421][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-shutdown]
[2023-04-26T02:01:20,421][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2023-04-26T02:01:20,422][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-stack]
[2023-04-26T02:01:20,422][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-text-structure]
[2023-04-26T02:01:20,422][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2023-04-26T02:01:20,422][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2023-04-26T02:01:20,422][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2023-04-26T02:01:23,062][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [1.1tb], net total_space [1.9tb], types [xfs]
[2023-04-26T02:01:23,062][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [7.8gb], compressed ordinary object pointers [true]
[2023-04-26T02:01:24,765][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [QZfXbrEpSretLKsQA3Yrnw], cluster name [my-application], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluste$
[2023-04-26T02:01:28,127][INFO ][o.e.x.s.Security         ] [node-1] Security is disabled
[2023-04-26T02:01:28,202][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/14890] [Main.cc@123] controller (64 bit): Version 8.4.3 (Build 9c00cf51c9fea9) Copyright (c) 2022 Elasticsearch BV
[2023-04-26T02:01:28,707][INFO ][o.e.t.n.NettyAllocator   ] [node-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_pa$
[2023-04-26T02:01:28,734][INFO ][o.e.i.r.RecoverySettings ] [node-1] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-04-26T02:01:28,771][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [multi-node] and seed hosts providers [settings]
[2023-04-26T02:01:29,937][INFO ][o.e.n.Node               ] [node-1] initialized
[2023-04-26T02:01:29,938][INFO ][o.e.n.Node               ] [node-1] starting ...
[2023-04-26T02:01:29,979][INFO ][o.e.x.s.c.f.PersistentCache] [node-1] persistent cache index loaded
[2023-04-26T02:01:29,980][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [node-1] deprecation component started
[2023-04-26T02:01:30,106][INFO ][o.e.t.TransportService   ] [node-1] publish_address {172.31.93.118:9300}, bound_addresses {[::]:9300}
[2023-04-26T02:01:32,532][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-04-26T02:01:32,534][WARN ][o.e.c.c.ClusterBootstrapService] [node-1] this node is locked into cluster UUID [IQ_uY407Rw2DsPqPvck7yQ] but [cluster.initial_master_nodes] is set to [ip-172-31-3-15.ec2.internal]; remove this setting to avoid possible d$
[2023-04-26T02:01:32,668][INFO ][o.e.c.s.MasterService    ] [node-1] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {node-1}{QZfXbrEpSretLKsQA3Yrnw}{54je2PlTSdiyLh0-hMaPgg}{node-1}{172.31.93.118}{172.31.93.118:9300}{cdfhilmrstw} completing elec$
[2023-04-26T02:01:32,892][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{QZfXbrEpSretLKsQA3Yrnw}{54je2PlTSdiyLh0-hMaPgg}{node-1}{172.31.93.118}{172.31.93.118:9300}{cdfhilmrstw}]}, term: 31, version: 90$
[2023-04-26T02:01:32,929][INFO ][o.e.r.s.FileSettingsService] [node-1] starting file settings watcher ...
[2023-04-26T02:01:32,937][INFO ][o.e.r.s.FileSettingsService] [node-1] file settings service up and running [tid=52]
[2023-04-26T02:01:32,941][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {172.31.93.118:9200}, bound_addresses {[::]:9200}
[2023-04-26T02:01:32,941][INFO ][o.e.n.Node               ] [node-1] started {node-1}{QZfXbrEpSretLKsQA3Yrnw}{54je2PlTSdiyLh0-hMaPgg}{node-1}{172.31.93.118}{172.31.93.118:9300}{cdfhilmrstw}{xpack.installed=true, ml.machine_memory=16776048640, ml.max_jvm$
[2023-04-26T02:01:33,004][WARN ][r.suppressed             ] [node-1] path: /my_newest/_search, params: {index=my_newest}
rg.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:176) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:162) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:935) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:769) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:390) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:478) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:277) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:103) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:186) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:81) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:121) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:119) ~[?:?]
rg.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:176) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:162) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:935) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:769) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:390) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:478) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:277) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:103) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:186) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:81) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:121) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.4.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:119) ~[?:?]

stephenb · April 26, 2023, 5:04am

Sorry but that is not the correct part of the logs... there are many many logs perhaps take the file

grep -i geo my-application.log

and show those results...

I am asking internally... I am seeing some inconsistency in 8.7.0... but oddly not with your version the geo dbs loaded right away...

BTW this is what geostats should look like... seeing if there is a command to force reload..

curl 'localhost:9200/_ingest/geoip/stats?pretty'
{
  "stats" : {
    "successful_downloads" : 3,
    "failed_downloads" : 0,
    "total_download_time" : 5082,
    "databases_count" : 3,
    "skipped_updates" : 0,
    "expired_databases" : 0
  },
  "nodes" : {
    "3-C7p0V6TZegryYhcK7ayA" : {
      "databases" : [
        {
          "name" : "GeoLite2-City.mmdb"
        },
        {
          "name" : "GeoLite2-Country.mmdb"
        },
        {
          "name" : "GeoLite2-ASN.mmdb"
        }
      ],
      "files_in_temp" : [
        "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb_LICENSE.txt",
        "GeoLite2-City.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb",
        "GeoLite2-City.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb",
        "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-Country.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb",
        "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
        "GeoLite2-Country.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb_README.txt"
      ]
    }
  }
}

I have one other thing you can try...

Try to disable then reenable

Disable the geoip databases

PUT _cluster/settings
{
  "persistent": {
    "ingest.geoip.downloader.enabled" : false
  }
}

GET _cat/indices/.ge*?v

GET _ingest/geoip/stats

Wait about 2 mins then re-enable

PUT _cluster/settings
{
  "persistent": {
    "ingest.geoip.downloader.enabled" : true
  }
}

GET _cat/indices/.ge*?v

GET _ingest/geoip/stats

Let me know if that works

You also try setting false / true in elasticsearch.yml and start stop see if that works

Akjal · April 26, 2023, 12:44pm

here is the log as you suggested after inputting grep -i geo my-application.log :

org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:134) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48) ~[?:?]
[2023-04-26T02:01:39,499][WARN ][o.e.i.g.GeoIpDownloader  ] [node-1] could not delete old chunks for geoip database [GeoLite2-Country.mmdb]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.deleteOldChunks(GeoIpDownloader.java:200) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.lambda$cleanDatabases$4(GeoIpDownloader.java:295) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.cleanDatabases(GeoIpDownloader.java:302) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:280) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.geoip_databases][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
[2023-04-26T02:01:40,508][WARN ][o.e.i.g.GeoIpDownloader  ] [node-1] could not delete old chunks for geoip database [GeoLite2-City.mmdb]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.deleteOldChunks(GeoIpDownloader.java:200) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.lambda$cleanDatabases$4(GeoIpDownloader.java:295) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.cleanDatabases(GeoIpDownloader.java:302) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:280) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102) ~[?:?]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.geoip_databases][0]]. Consider using `allow_partial_search_results` setting to bypass this error.

this is the same output as before when i run curl 'localhost:9200/_ingest/geoip/stats?pretty'

{
  "stats" : {
    "successful_downloads" : 0,
    "failed_downloads" : 1,
    "total_download_time" : 0,
    "databases_count" : 0,
    "skipped_updates" : 0,
    "expired_databases" : 3
  },
  "nodes" : { }
}

Akjal · April 26, 2023, 1:34pm

Thank you so much @stephenb !!!, , it is working now!, I disabled and reanabled the geoip.downloader as you outlined above and it worked. Miraculous

system · May 24, 2023, 3:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.