Geoip2 ASN resolution in Ingest Pipeline

Elastic Stack Elasticsearch
1

Hello, I tried to use GeoIP2 plugin to resolve iP to ASN. Unfortunately during log processing I have information in tag:

	
[netflow, forwarded, _geoip_database_unavailable_GeoLite2-City.mmdb, _geoip_database_unavailable_GeoLite2-City.mmdb, _geoip_database_unavailable_GeoLite2-ASN.mmdb, _geoip_database_unavailable_GeoLite2-ASN.mmdb]

I am trying to read data from netflow.destination_ipv4_address.

    {
      "geoip": {
        "field": "netflow.destination_ipv4_address",
        "target_field": "destination.as",
        "database_file": "GeoLite2-ASN.mmdb",
        "properties": [
          "asn",
          "organization_name"
        ],
        "ignore_missing": true
      }
    },

Could you help me why database are not avaiable?

I tried to disable plugin:

PUT _cluster/settings
{
  "persistent": {
    "ingest.geoip.downloader.enabled" : false
  }
}

Next I restarted all nodes, changed parametr to true and restarted nodes. But It didn't help me

Error logs:

[2024-03-25T12:28:53,361][ERROR][o.e.i.g.GeoIpDownloader  ] [node-1] error downloading geoip database [GeoLite2-ASN.mmdb]
   at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.processDatabase(GeoIpDownloader.java:179) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:161) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:60) ~[?:?]
2

If you want to use the GeoIP processor you need this to be enabled.

Your nodes have internet access? What is the return of the following request on dev tools?

GET _ingest/geoip/stats
3

Thank you for your anserw. I set temporary this option to false, after restart I change again to true and restart.

Unfortunately I have a error:

Request:
GET _ingest/geoip/stats

Response:
{
  "stats": {
    "successful_downloads": 0,
    "failed_downloads": 3,
    "total_download_time": 0,
    "databases_count": 0,
    "skipped_updates": 0,
    "expired_databases": 0
  },
  "nodes": {}
}

I log file I founded:

[2024-03-25T12:30:55,750][ERROR][o.e.i.g.GeoIpDownloader  ] [node-1] error downloading geoip database [GeoLite2-ASN.mmdb]
   at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.processDatabase(GeoIpDownloader.java:179) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:161) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:60) ~[?:?]
[2024-03-25T12:30:55,780][ERROR][o.e.i.g.GeoIpDownloader  ] [node-1] error downloading geoip database [GeoLite2-City.mmdb]
   at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.processDatabase(GeoIpDownloader.java:179) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:161) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:60) ~[?:?]
[2024-03-25T12:30:55,791][ERROR][o.e.i.g.GeoIpDownloader  ] [node-1] error downloading geoip database [GeoLite2-Country.mmdb]
   at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
   at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.processDatabase(GeoIpDownloader.java:179) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:161) ~[?:?]
   at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:60) ~[?:?]
4

Did you validate that? Check if the nodes can access https://geoip.elastic.co/v1/database

Use a curl from the node to test.

https://geoip.elastic.co/v1/database

You should get a 400 response, because you are not sending the correct payload, but this would test if the node has internet connection or not to the endpoint.

5

Yes I tried that:

root@siem:# curl https://geoip.elastic.co/v1/database
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>400 Bad Request</title>
</head>
<body align="center">
    <div role="main" align="center">
        <h1>400: Bad Request</h1>
        <p>The request could not be understood by the server due to malformed syntax.</p>
        <hr />
    </div>
    <div role="contentinfo" align="center">
        <small>Rocket</small>
    </div>
</body>
6

Hi @Patryk_Ostrowski

There was another user having issues... see below

What version are you on?

You could set the logs to TRACE and see if there's more detail

You can use this

PUT /_cluster/settings
{
  "persistent": {
    "logger.org.elasticsearch.ingest.geoip": "TRACE"
  }
}

You can turn off with

PUT /_cluster/settings
{
  "persistent": {
    "logger.org.elasticsearch.ingest.geoip": null
  }
}

Then Turn off / on the etc

PUT _cluster/settings
{
  "transient": {
    "ingest.geoip.downloader.enabled": false
  }
}

PUT _cluster/settings
{
  "transient": {
    "ingest.geoip.downloader.enabled": true
  }
}

PUT _cluster/settings
{
  "transient": {
    "ingest.geoip.downloader.eager.download": true
  }
}

The downloader temporarily downloads the files to /tmp which needs to be writeable by the elasticsearch user or perhaps it's out of space

This thread goes through some of the things we tried.

If you read up this thread, you'll see some of the things that we tried.

In the end he loaded manually.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.