How to use custom GeoIP2 database in elasticsearch?

[icamys (Dmitrii)]

Hello!

I'm using dockerized elasticsearch7.3 on my local machine and I'm trying to use my GeoIP2-City.mmdb to add geoip info.

I've read the length and breadth of the official geoip processor description and I still can't manage to create a processor with custom geoip library.

I'm sending HTTP PUT request to

http://{{ elasticsearch_host }}:{{ elasticsearch_port }}/_ingest/pipeline/geoip

with data:

{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "ip",
        "target_field" : "geo",
        "database_file" : "GeoIP2-City.mmdb"
      }
    }
  ]
}

And getting the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "[database_file] database file [GeoIP2-City.mmdb] doesn't exist",
        "property_name": "database_file",
        "processor_type": "geoip"
      }
    ],
    "type": "parse_exception",
    "reason": "[database_file] database file [GeoIP2-City.mmdb] doesn't exist",
    "property_name": "database_file",
    "processor_type": "geoip"
  },
  "status": 400
}

However if I try to create pipeline with database_file field set to the name of any shipped libraries (ex. GeoLite2-City.mmdb), the pipeline is created successful.

What I've done until now:

1. Created Dockerfile for elasticsearch:

    ARG ES_VERSION=7.3.0
    FROM docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION}
    
    # Here I copy the Geoip library and set the same ownership
    # as other geoip lite libraries have.
    COPY ./libs/GeoIP2-City.mmdb /usr/share/elasticsearch/modules/ingest-geoip/GeoIP2-City.mmdb
    RUN chown elasticsearch:root /usr/share/elasticsearch/modules/ingest-geoip/GeoIP2-City.mmdb
   

   Here I've supposed that mentioned in docs $ES_CONFIG/ingest-geoip is the /usr/share/elasticsearch/modules/ingest-geoip (also I've tried /usr/share/elasticsearch/config/ingest-geoip) because there are absolutely no mentions in docs about the $ES_CONFIG variable.

2. Starting elasticsearch as a service via docker-compose:

    elasticsearch:
      build:
        context: ./elasticsearch
        dockerfile: Dockerfile
        args:
          ES_VERSION: 7.3.0
      network_mode: host
      env_file:
        - ./elasticsearch/env.sh
      volumes:
        - ./data/elastic:/usr/share/elasticsearch/data
   

Any help is appreciated.

Thank you.

[spinscale (Alexander Reelsen)]

Hey,

see this snippet from the geoip processor docs at GeoIP Processor | Elasticsearch Guide [7.3] | Elastic

The geoip processor can run with other GeoIP2 databases from Maxmind. The files must be copied into the ingest-geoip config directory, and the database_file option should be used to specify the filename of the custom database. Custom database files must be stored uncompressed. The ingest-geoip config directory is located at $ES_CONFIG/ingest-geoip .

It does not look as if the file has been copied into the config directory in your docker file.

[icamys (Dmitrii)]

Hi! Thanks for your response!

I don't know where the config directory is exactly located (the regular /etc/elasticsearch does not exist inside the elastcisearch container, the $ES_CONFIG is not set) that's why I've tried to copy the database inside the directories:

• /usr/share/elasticsearch/modules/ingest-geoip/
• /usr/share/elasticsearch/config/ingest-geoip/
• /usr/share/elasticsearch/config/
• /etc/elasticsearch/ingest-geoip/

None of those worked. Any ideas?

[spinscale (Alexander Reelsen)]

/usr/share/elasticsearch/config/ingest-geoip sounds good to me. can you share your dockerfile in a gist in order to reproduce?