JSON latitude/longitude to geo_point

Elastic Stack Elasticsearch
1

I'm looking for some guidance or an example on creating a geo_point from JSON latitude and longitude fields which I can then visualize in Kibana 4 on a Tile Map. I'm in control of the data output so I can format it any which way before Logstash processes it. Some of the examples I've seen involve mutating and converting fields which I've attempted below. This results in a "location" field in Elasticsearch with a [lon,lat] format and is displayed in Kibana as a "number" datatype. I'm unable to visualize this as a geo_point on a Tile Map, however. I'm assuming this is because the field is mapped to a number and not a geo_point.

Here's a snippet of what the JSON looks like (I've omitted other fields for brevity):

{  
   "@timestamp":"2015-06-03T22:20:44.000Z",
   "latitude":39.613658,
   "longitude":-86.106653,
   "location":[-86.106653,39.613658]
}

My logstash config (omitted certain values for sensitivity):

input {
  pipe {
    command => "..."
    type => "..."
    codec => "json"
  }
}
filter {
  json {
    source => "message"
    target => "tweet"
  }
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
  if [latitude] and [longitude] {
    mutate {
	  add_field => [ "[location]", "%{longitude}" ]
	  add_field => [ "[location]", "%{latitude}" ]
  }
  mutate {
    convert => [ "[location]", "float" ]
  }
}  
output {
  elasticsearch {
    host => "..."
    protocol => "http"
    cluster => "..."
    user => "..."
    password => "..."
    index => "logstash-twitter-%{+YYYY.MM.dd}"
  }
}

How should I map the location field to a geo_point datatype?

I'm certain others have run into similar issues and maybe there's a simple solution that I'm not realizing or seeing online. I'm fairly new to the ELK stack so I appreciate the help. Can someone please point me in the right direction?

Thanks!

2

You need to set the mapping manually in ES.
Take a look at this thread for somes hint - Kibana 4 and geoip

3

Thank you for your reply, Mark, that was helpful. For others experiencing the same issue, here's what I did (because I couldn't really find a detailed guide on this):

Deleted my existing Logstash index template from Elasticsearch:

curl -XDELETE 'http://localhost:9200/_template/logstash'

Modified the following template JSON file (make sure to back it up first):

/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.4-java/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

In the "properties" section, I added "location" and made it a "geo_point" type (note: geoip already existed):

"properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         },
         "location" : { "type": "geo_point" }
       }

In Kibana, I deleted the existing logstash index pattern and re-created it. Now the "location" field is mapped to a "geo_point" and I'm able to visualize it on the Tile Map.

Thanks again!

4

"properties" : {
"@timestamp": { "type": "date", "doc_values" : true },
"@version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"ip": { "type": "ip", "doc_values" : true },
"location" : { "type" : "geo_point", "doc_values" : true },
"latitude" : { "type" : "float", "doc_values" : true },
"longitude" : { "type" : "float", "doc_values" : true }
}
},
"location": {
"type": "geo_point"
}
}

I am using like this and I deleted my _template/logstash. After that I view the _template/logstash using curl -XGET

"properties": {
"@timestamp": {
"doc_values": true,
"type": "date"
},
"location": {
"type": "geo_point"
},
"geoip": {
"dynamic": true,
"properties": {
"location": {
"doc_values": true,
"type": "geo_point"
},
"longitude": {
"doc_values": true,
"type": "float"
},
"latitude": {
"doc_values": true,
"type": "float"
},
"ip": {
"doc_values": true,
"type": "ip"
}
},
"type": "object"
},
"@version": {
"index": "not_analyzed",
"doc_values": true,
"type": "string"
}
},
"_all": {
"enabled": true,
"omit_norms": true
}

Kibana it is still shows as string
location.lon string
Edit
location.lat string

5

Please start your own thread, this one is a year old!