How to use custom GeoIP2 database in elasticsearch?

Elastic Stack Elasticsearch
1

Hello!

I'm using dockerized elasticsearch7.3 on my local machine and I'm trying to use my GeoIP2-City.mmdb to add geoip info.

I've read the length and breadth of the official geoip processor description and I still can't manage to create a processor with custom geoip library.

I'm sending HTTP PUT request to

http://{{ elasticsearch_host }}:{{ elasticsearch_port }}/_ingest/pipeline/geoip

with data:

{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "ip",
        "target_field" : "geo",
        "database_file" : "GeoIP2-City.mmdb"
      }
    }
  ]
}

And getting the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "[database_file] database file [GeoIP2-City.mmdb] doesn't exist",
        "property_name": "database_file",
        "processor_type": "geoip"
      }
    ],
    "type": "parse_exception",
    "reason": "[database_file] database file [GeoIP2-City.mmdb] doesn't exist",
    "property_name": "database_file",
    "processor_type": "geoip"
  },
  "status": 400
}

However if I try to create pipeline with database_file field set to the name of any shipped libraries (ex. GeoLite2-City.mmdb), the pipeline is created successful.

What I've done until now:

  1. Created Dockerfile for elasticsearch:

     ARG ES_VERSION=7.3.0
 FROM docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION}
 
 # Here I copy the Geoip library and set the same ownership
 # as other geoip lite libraries have.
 COPY ./libs/GeoIP2-City.mmdb /usr/share/elasticsearch/modules/ingest-geoip/GeoIP2-City.mmdb
 RUN chown elasticsearch:root /usr/share/elasticsearch/modules/ingest-geoip/GeoIP2-City.mmdb

    Here I've supposed that mentioned in docs $ES_CONFIG/ingest-geoip is the /usr/share/elasticsearch/modules/ingest-geoip (also I've tried /usr/share/elasticsearch/config/ingest-geoip) because there are absolutely no mentions in docs about the $ES_CONFIG variable.

  2. Starting elasticsearch as a service via docker-compose:

     elasticsearch:
   build:
     context: ./elasticsearch
     dockerfile: Dockerfile
     args:
       ES_VERSION: 7.3.0
   network_mode: host
   env_file:
     - ./elasticsearch/env.sh
   volumes:
     - ./data/elastic:/usr/share/elasticsearch/data

Any help is appreciated.

Thank you.

2

Hey,

see this snippet from the geoip processor docs at GeoIP Processor | Elasticsearch Guide [7.3] | Elastic

The geoip processor can run with other GeoIP2 databases from Maxmind. The files must be copied into the ingest-geoip config directory, and the database_file option should be used to specify the filename of the custom database. Custom database files must be stored uncompressed. The ingest-geoip config directory is located at $ES_CONFIG/ingest-geoip .

It does not look as if the file has been copied into the config directory in your docker file.

3

Hi! Thanks for your response!

I don't know where the config directory is exactly located (the regular /etc/elasticsearch does not exist inside the elastcisearch container, the $ES_CONFIG is not set) that's why I've tried to copy the database inside the directories:

  • /usr/share/elasticsearch/modules/ingest-geoip/
  • /usr/share/elasticsearch/config/ingest-geoip/
  • /usr/share/elasticsearch/config/
  • /etc/elasticsearch/ingest-geoip/

None of those worked. Any ideas?

4

/usr/share/elasticsearch/config/ingest-geoip sounds good to me. can you share your dockerfile in a gist in order to reproduce?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.