Geopoint visualization in Kibana 6.4.2 (Elastic 6.4.2)

Elastic Stack Kibana
9

So here is a postmortem of my efforts to correctly show geo_point's in my visualization

  1. I have real time data coming in through a REST API - this contains a "points" query param
  2. The data is ingested by a filebeat plugin which sends it to logstash
  3. Logstash's filter plugin parses and mutates it to an an object with 2 floating point values
  4. Logstash's output plugin writes it to a filebeat index on ES
  5. ES, Logstash and Kibana are all version 6.4
  6. Filebeat indexes are created via a script and there is a new index every day e.g. for today's it's called filebeat-2019.11.20

So I finally did the following, in order for the geo_point's to show up correctly

  1. I updated the mapping to my latest filebeat by sending
    curl -X PUT "localhost:9200/filebeat-2019.11.20/_mapping/doc?pretty" -H 'Content-Type: application/json' -d' { "properties": { "req_gp": { "type": "geo_point" } } } '
    Please note I am using "doc" as the type name instead of "_doc" since Logstash 6.4 auto creates a type called "doc" - the PUT Mapping reference for 6.4 on ES's official docs mention _doc and updating the mapping for this type throws an error

Rejecting mapping update to [<index_name>] final mapping would have more than 1 type: [_doc, doc]"

  1. Next I changed the field name in logstash's filter plugin:

       mutate {                                                                                                                                                                                                             
     split => { "point" => ";" }                                                                                                                                                                                      
   }                                                                                                                                                                                                            
   mutate {
     add_field => {
       "[req_gp][lat]" => "%{[point][0]}"
       }

     add_field => {
       "[req_gp][lon]" => "%{[point][1]}"
       }
   }

   mutate {
     convert => { "[req_gp][lat]" => "float" }
     convert => { "[req_gp][lon]" => "float" }
   }

I created a new field since you cannot change the mapping for an existing field in ES

Also, please note I created the mapping in the index before sending the data to this new field, otherwise ES would save the mapping as {float, float} for this field and again, it cannot be changed later

  1. Lastly, I restarted Logstash

  2. Updated the index pattern in Kibana and made sure only the field itself is listed with a "geopoint" type

  3. Clicked "Visualize" next to the field name on the filter on the left and it showed up successfully in the Map Coordinates visualization

Later on I intend to all the new field mapping to previous indices and run update_by_query to move the existing data in the old indices to the new type in the same indices

P.S. Big thanks to Joe Reuter for helping out!