Unable to get geoip.location to a geo_point in ES

Elastic Stack Elasticsearch
1

I've read dozens of posts about doing this but am still unable to get a location into ES as a geo_point. I am using the following logstash conf snippet:

input { stdin {}}
filter { geoip { source => "message" } }
output {
  stdout { codec => rubydebug }
    elasticsearch {
      hosts => "elk.stolaf.edu:9200"
      template_name => "testtemplate"
      index => "test-%{+YYYY.MM.dd}"
    }
  }

I enter an IP and see the following JSON document:

{
      "@version" => "1",
          "host" => "logs",
    "@timestamp" => 2017-11-10T21:07:24.269Z,
         "geoip" => {
              "timezone" => "Europe/Paris",
                    "ip" => "83.204.73.133",
              "latitude" => 46.9377,
        "continent_code" => "EU",
             "city_name" => "Rocheserviere",
          "country_name" => "France",
         "country_code2" => "FR",
         "country_code3" => "FR",
           "region_name" => "Vendée",
              "location" => {
            "lon" => -1.5114,
            "lat" => 46.9377
        },
           "postal_code" => "85620",
           "region_code" => "85",
             "longitude" => -1.5114
    },
       "message" => "83.204.73.133"
}

So far so good. I retrieved the mapping for this newly created index and create a template updating the location to a geo_point. Here is the template I upload (curl -XPUT 'http://localhost:9200/_template/testtemplate?pretty' -d @/tmp/testtemplate.json):

{
  "testtemplate": {
    "order": 0,
    "version": 50001,
    "template": "test-*",
    "settings": {
      "index": {
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "_default_": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "geoip": {
            "properties": {
              "city_name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
             < some entries omitted to stay within post length >
              "latitude": {
                "type": "float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "float"
              },
             < some entries omitted to stay within post length >
            "host": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "message": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

I delete the index and restart my test logstash config. I paste in the same IP and get the same JSON. When I retrieve the mapping for the index, it appears to have added a second type "logs". (I omitted this to stay within post limits.) Each type in the mapping contains: "location": { "type": "geo_point" }.

When I try to build a map in Kibana it tells me "Index pattern does not contain any of the following field types: geo_point". When I look in Management > Index Patterns for the test-* index, I see separate entries for geoip.location.lat and geoip.location.lon, and both are type number.

Thanks!!

2

Can you show the mapping for the index? Feel free to use gist/pastebin/etc :slight_smile:

3

https://gist.github.com/anonymous/cd518e4b0a311fede07356cc0b1b94ff

4

Looks good.
What does a doc from that same index look like?

5 
[user@elk ~]# curl -XGET 'http://localhost:9200/test-2017.11.10/_search?q=geoip.ip:83.204.73.133&pretty=true'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "test-2017.11.10",
        "_type" : "logs",
        "_id" : "AV-nwjyENlMqofZI4Icj",
        "_score" : 0.2876821,
        "_source" : {
          "@version" : "1",
          "host" : "logs",
          "@timestamp" : "2017-11-10T21:07:24.269Z",
          "geoip" : {
            "timezone" : "Europe/Paris",
            "ip" : "83.204.73.133",
            "latitude" : 46.9377,
            "continent_code" : "EU",
            "city_name" : "Rocheserviere",
            "country_name" : "France",
            "country_code2" : "FR",
            "country_code3" : "FR",
            "region_name" : "Vendée",
            "location" : {
              "lon" : -1.5114,
              "lat" : 46.9377
            },
            "postal_code" : "85620",
            "region_code" : "85",
            "longitude" : -1.5114
          },
          "message" : "83.204.73.133"
        }
      }
    ]
  }
}
6

And you've done a refresh of the mappings for that index pattern in Kibana? And if so it still shows not geopoint?

7

And you've done a refresh of the mappings for that index pattern in Kibana?

Facepalm. That did it. Thanks!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.