Geo point mapping

Elastic Stack Logstash
1

Hi,

my dataset basically consists of the geo-information, but the format is not as perfect as I can directly ingest them in and let ES recognize it as the geo point type.

{"geo_information": "united states/michigan/holt:[42.638,-84.522]"}

This is the original format. I am wondering if there is any preprocessing way in logstash to exact the lat and lon information from the original value. And make it turn into the geo point type.

P.S What I want is to keep the original data information and add more fields based on the original data information.

Something like:

{"geo_information": "united states/michigan/holt:[42.638,-84.522]"}
{“geo_lon” : 42.638}
{"geo_lat" : ...}

I did go through the document of geo-point, but kinda confused how to actually operate it in my specific case.

Thanks.

2

You need to use grok or dissect to pull it apart and then assign it as lat + lon fields. There's nothing natively in Logstash to do this with your format.

3

Thanks.

And also if I get two more fields like
{“geo_lon” : 42.638}
{"geo_lat" : ...}

How does ES recognize the type of those fields is the regular float or geo point. My goal is to make it be geo point.

4

They need to be one of the structures as defined here - https://www.elastic.co/guide/en/elasticsearch/reference/6.2/geo-point.html - and then the resultant field needs to be mapped accordingly.

5

This is my updated config,

	grok{
		match => {[geo_information] => "%{GREEDYDATA}%{NUMBER:lat:float}%{GREEDYDATA}%{NUMBER:lon:float}%{GREEDYDATA}"}
		add_filed => {"location" => "[[lat],[lon]]"}
	}

When I actually ran it, it threw me

Grok regexp threw exception {:exception=>"no implicit conversion of Array into String", :backtrace=>["/Users/apple/Desktop/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.2/lib/logstash/filters/grok.rb:320:in match'", "/Users/apple/Desktop/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.2/lib/logstash/filters/grok.rb:296:inblock in filter'", "org/jruby/RubyHash.java:1343:in each'", "/Users/apple/Desktop/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.2/lib/logstash/filters/grok.rb:295:infilter'", "/Users/apple/Desktop/logstash-6.2.2/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "/Users/apple/Desktop/logstash-6.2.2/logstash-core/lib/logstash/filters/base.rb:164:inblock in multi_filter'", "org/jruby/RubyArray.java:1734:in each'", "/Users/apple/Desktop/logstash-6.2.2/logstash-core/lib/logstash/filters/base.rb:161:inmulti_filter'", "/Users/apple/Desktop/logstash-6.2.2/logstash-core/lib/logstash/filter_delegator.rb:47:in multi_filter'", "(eval):202:inblock in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):198:inblock in initialize'", "(eval):148:in block in filter_func'", "/Users/apple/Desktop/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:447:infilter_batch'", "/Users/apple/Desktop/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:426:in worker_loop'", "/Users/apple/Desktop/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:385:inblock in start_workers'"], :class=>"TypeError"}

And I noticed the example in the document, they usually do the match with "message", I am not sure if it is available for us to do the filter parsing with the specific field, like "geo_information" in my case.

6

Is that initial example a complete event/message?

7

Nah, the complete message is
{"geo_information": "....", "name": ".....", "age": "......"}

Actually, my message is more complicated than this. For the clear visualization, I did the cutting of the whole message in this scenario.

8

do you mean add_field?

I would advise coming up with a pattern that matches the shape of your data a little more explicitly; GREEDYDATA is very, very greedy (as the name implies), and you run the risk of it capturing more than you intend. If your pattern is always :[ (latitude) , (longitude) ], we can get pretty specific (note: the square brackets need to be prefixed by a backslash to escape them, since square brackets normally have special meaning in regular expressions and grok patterns):

":\[%{NUMBER:geo_lat:float},%{NUMBER:geo_lon:float}\]"
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.