Converting to geo_point

Elastic Stack Logstash
1

Hi,

For now, I have my own fields called "lat" and "lon" in the json file.

I was trying to make ES recognize it as the geo_point data so I did some kinds of editing in Logstash.

filter {
	mutate{
		add_field => {"location" => [[lat],[lon]]}
	}
}

But it turned out that the type of geoip.location is not geo_point.

I am wondering if there is something wrong with my filter?

Updated: the logstash threw me the error like this

[2018-03-23T20:04:37,029][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"locationdata", :_type=>"eventdata", :_routing=>nil}, #<LogStash::Event:0x6bfc6277>], :response=>{"index"=>{"_index"=>"locationdata", "_type"=>"eventdata", "_id"=>"XNFSVWIB-SahT3BDQlYN", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"illegal latitude value [269.9986267089844] for location"}}}}}
{
              "lon" => 25,
       "@timestamp" => 2018-03-24T00:04:36.649Z,
             "host" => "appletekiMacBook-Pro.local",
         "@version" => "1",
              "lat" => 15,
    "location_info" => "united/michigan:[15,25]",
         "location" => [
        [0] "[\"lat\"]",
        [1] "[\"lon\"]"
    ],
             "path" => "/Users/apple/Desktop/Location/location.json"
}
2

There are a number of accepted forms for geo_point; I believe you were going with the two-element array variant (e.g., [12.34, 56.78]), but it ends up being easier to use the object variant (e.g., {"lat": 12.34, "lon": 56.78}).

Since the add_field directive can only add string values, if we wanted to use it we would need to do some extra work to convert the values back to numeric values; instead, we can use rename to move both values (or copy to copy them and leave the previous fields in-tact)

filter {
  mutate {
    rename => {
      "lat" => "[location][lat]"
      "lon" => "[location][lon]"
    }
  }
}

OR

filter {
  mutate {
    copy => {
      "lat" => "[location][lat]"
      "lon" => "[location][lon]"
    }
  }
}

If you do need the two-element-array syntax for some reason (it makes no difference to Elasticsearch which one we use as long as the field is defined as a geo_point in the index mapping), we would have to change it up a bit.

The syntax [[lat],[lon]] that you have isn't quite right to get us there; it parses out to mean

  • an array, containing:
    • an array, containing the literal string lat; AND
    • an array, containing the literal string lon

Instead, we need to add a two-element array, containing the value-at-[lat] and the value-at-[lon], and since this produces strings, we would need to convert them back to floating-point numbers:

filter {
  mutate {
    add_field => {
      "location" => ["%{[lat]}","%{[lon]}"]
    }
  }
  mutate {
    convert => {
      "location" => "float"
    }
  }
}

For good measure, if we wanted the string variant, it would be pretty easy too:

filter {
  mutate {
    add_field => {
      "location" => "%{lat},%{lon}"
    }
  }
}
3

Thank you so much. Your demonstration helped me a lot.

I have some questions tho. The first is if I just used your last method, ES will recognize "location" as the type of string. So do I need to set the mapping for this data to specify the type of "location" should be geo_point in the console of kibana in advance? I am kinda confused cuz I thought ES will automatically do the recognizion.

For getting the value of the field, you mentioned using %{[lat]}. I am wondering if it works for the case that I want to parse something from the specific field which is in the original input data.

Or could you take a look at Geo point mapping

Thanks again!!!

4

yes, it is advised to set up the index mapping, either explicitly or via a template.

Any time we make a computer "guess" a best fit, we run the risk of it guessing incorrectly, so it's best to be explicit wherever possible.

The add_field directive, and many others, allow us to reference existing fields from the event by name using what is called sprintf notation; not all directives support it, so you'll need to rely on documentation for the particular plugin and/or experimentation.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.