SOLVED - Long & Lat Help - I've been reading many posts & still no answers

[Wayne_Taylor]

Team,

My data source has Longtitude and Latitude in it. I have time series data of aircraft and want to plot the map progress.

I am using Logstash 5.3.0 from an event stream. Below is my configuration. Currently when the data is ingested into Elasticsearch its still showing as String (1st attempt), Numeric (2nd attempt and many after that). I want to get to a point I can use maps in Kibana.

Filter Section:

mutate {
  convert => { "longitude" => "float" }
  convert => { "latitude" => "float" }
}
    
mutate {
  add_field => { "[location][lat]" => "%{latitude}" }
  add_field => { "[location][lon]" => "%{longitude}" }
}
            
date {
  match => [ "time_at_position", UNIX_MS ]
  target => "@timestamp"
}

My Output Section (excluded user details and host:

elasticsearch {
  index => "flightpostest"
  document_type => "positional"
  manage_template => "false"
  template_name => "positional"
}

My Template:

PUT _template/positional
{
  "template": "positional*",
  "settings": {},
  "mappings": {
    "_default_": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}

What am I missing to get this as a Geo Point so I can use Maps. I've read this: https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html
and the engineering post and blog but still stumped.

Please help.

Wayne

[Badger]

Doing the convert of longitude and latitude has no effect on the geo_point. add_field converts them back to string. You could do a convert on "[location][lat]" but you do not need to. geo_point will work with either string or float.

Get rid of the template and create the index using

PUT flightpostest
{
  "mappings": {
    "doc": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}

Then in your output just name the index

elasticsearch {
index => "flightpostest"
hosts => [ "localhost" ]
}

[Wayne_Taylor]

@Badger - I made the changes as per suggestion and now getting the following exception:

[2018-03-01T23:17:27,421][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"flightpostest", :_type=>"positional", :_routing=>nil}, #LogStash::Event:0x7cda5271], :response=>{"index"=>{"_index"=>"flightpostest", "_type"=>"positional", "_id"=>"AWHj2yweXQkqogeoXm0O", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[location] is defined as an object in mapping [positional] but this name is already used for a field in other types"}}}}

[Badger]

@Wayne_Taylor

now getting the following exception

I banged my head on that one for a while. Did you delete the template that you previously added? I think that is what fixed it for me.

[Wayne_Taylor]

@Badger - error gone now but back to strings :(.

[wwalker (Walker)]

Throw my two cents in here. Instead of expressing my geo_point as an object, I have it configured as a string, example 2 in the reference manual. You may want to give it a shot, if you haven't already, it appears as though it may be a lot less work for your pipeline.

add_field => { "[geoip][location][coordinates]" => "%{[geoip][location][lat]}, %{[geoip][location][lon]}" }

Just to be clear, my field names are geoip.location.coordinates, geoip.location.lat, and geoip.location.lon; hence the reason for all the extra brackets...can be confusing if you've never encountered it before.

[Badger]

If I create an index using

PUT differentname
{
  "mappings": {
    "doc": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}

and populate it using this

input { generator { message => '{ "longitude" : -1.2 , "latitude" : 3.45, "t": 1492310893103, "foo" : 1 }' count => 1 } }
output { stdout { codec => rubydebug } }
filter {
  json { source => "message" }
  mutate {
    add_field => { "[location][lat]" => "%{latitude}" }
    add_field => { "[location][lon]" => "%{longitude}" }
  }
                
  date {
    match => [ "t", UNIX_MS ]
    target => "@timestamp"
  }
}
output {
  elasticsearch {
    index => "differentname"
    hosts => [ "localhost" ]
  }
}

then I get a geo_point. Does that work for you, or do you still get the conflict?

[Wayne_Taylor]

@Badger - i get conflict

[2018-03-02T18:34:38,900][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"positional", :_type=>"positional", :_routing=>nil}, #LogStash::Event:0x76c34393], :response=>{"index"=>{"_index"=>"positional", "_type"=>"positional", "_id"=>"AWHn_p2lXQkqogeocS_l", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[location] is defined as an object in mapping [positional] but this name is already used for a field in other types"}}}}

[Badger]

You used the index positional. Don't do that. Using the index name differentname was the point

[Wayne_Taylor]

Ok - but same issue

2018-03-02T19:04:17,379][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"differentname", :_type=>"GPS_EVENT", :_routing=>nil}, #LogStash::Event:0x228da33f], :response=>{"index"=>{"_index"=>"differentname", "_type"=>"GPS_EVENT", "_id"=>"AWHoGcAxXQkqogeocayS", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[location] is defined as an object in mapping [GPS_EVENT] but this name is already used for a field in other types"}}}}
[2018-03-02T19:04:17,881][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x5495491d run>"}
[

Why is this difficult

[Badger]

@Wayne_Taylor

Ok - but same issue

Then I am out of ideas. Sorry.

[Wayne_Taylor]

alright @Badger - got it working. It bugged the crap out of my that your dummy sample worked - but didn't work with my real config. I then did a lot of forum searching on the error message returned now and found this: Errors with geo_point.

From that, deleted my old index, put my mapping and then ingested and boom

What a pain in the ass.

Thank you so much for helping.

[hbj (Halldor)]

This is very interesting.

@Wayne_Taylor Could you share a sample of your input and your final config?

Very curios to see what the output looks like. I'm wanting to do the same - i.e. plot aircraft and/or even generate heatmaps.