Add geoip on existing index

[snicoll (Stéphane Nicoll)]

Hey,

I have an existing index that contains a "Request" type with namely an attribute with the IP of the user. That index holds several millions of documents. I'd like to use Kibana to get some stats about the traffic to that service.

The index is updated by some custom code, it is not related to logstash in any way. I've followed this article[1] that explains how to do it with logstash but I am wondering if it would be possible to apply the same pattern to an existing index.

Ideally, I'd like a background process to run on my index and add the geolocation information based on the IP. Is there some documentation or some plugin that would do that?

Thanks!

[1] https://www.elastic.co/blog/geoip-in-the-elastic-stack

[Christian_Dahlqvist]

You should be able to do that through the reindex API together with an ingest pipeline with a ingest-geoip processor.

[snicoll (Stéphane Nicoll)]

Awesome, thanks a lot!

In case someone is interested by this, here are the steps to make it working

• Install the ingest-geoip processor (sudo bin/elasticsearch-plugin install ingest-geoip)
• Create your pipeline (some document may not have the ip so I've tuned it so that it doesn't stop if one document doesn't have it). The IP is stored in requestIpv4 in my document:

PUT _ingest/pipeline/geoip
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "requestIpv4",
        "ignore_missing" : true
      }
    }
  ]
}

• Create a new index that is similar to the index you want to migrate (for instance foo-geo for your existing foo index) with an additional property in your mapping

"geoip": {
    "type": "object",
    "dynamic": true,
    "properties": {
        "location": {
            "type": "geo_point"
        }
    }
}

• Reindex your foo index to foo-geo, adding the geolocation information:

POST _reindex
{
  "source": {
    "index": "foo"
  },
  "dest": {
    "index": "foo-geo",
    "pipeline": "geoip"
  }
}

I guess the next step is to create the mapping on foo and move foo-geo to foo again. I haven't found an easier way to rename the index.