An almost related logstash geohash ignorance, yet far enough to have me lost

[drfuka (Daniel Fuka)]

Aloha,
I can not seem to figure out how to map a geohash that I am pulling in from logstash. A) I can not get logstash to stash it as a "geohash" or related type, and B) I can not figure out the shortest route to getting the geohash mapped in Kibana through any of the many possible ways.

I have a device sending a chirp of data to a webserver which I am capturing and parsing with logstash.

With grok filter I have tried every type I can find in the source code, including "geohash", "geohash_grid", "geo_point", ... yet it keeps creating the index as a string without geo_point/geohash type.

grok {
match => [ "request", "GO%{DATA:geohash:geo_point}HP" ]
}

t message 198.82.45.11 - - [24/May/2019:19:32:28 -0400] "GET /VTSensor?SV02DA19052tvs3227GOdqbtx4cqh400tzHP0HU1VK48V1:SA2,5345@38SP%29/86@32SA5,55/4@34SA6,45/61@28SA9<25/@19SA10=/253@19SA12,%5>273B34SA17,15/130~WA19,33/13$@29SA20,29/298@28SA235 HTTP/1.1" 404 448 "-" "curl/7.40.0"

And no matter what I try to set it as, it comes back as a string rather than geo_point:

t geohash dqbtx4cqh400tz

curl 'http://localhost:9200/logstash-*/_mapping'

---snip---

"geohash":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}}

Yes, this is a stupid question... please be kind enough to humiliate me with the shortest workflow to getting these geohashes plotted and I will buy you a round or two whenever we shall meet. {: -)

Thanks!
dan

[Christian_Dahlqvist]

I would recommend having a look at this blog post as it shows a similar type of event parsed using grok and dissect. The key here is understanding the difference between types in Logstash and Elasticsearch. The types you can specify or cast to in Logstash only determine how the fields are formatted in the generated JSON document, so a limited number of types are available (string, integer, float etc). Elasticsearch mappings define how the fields that come in from Logstash are indexed and these mappings have a lot more options and flexibility. A field formatted as string in JSON can be indexed as a date, ip address, keyword etc.

For geo-points you can therefore not do it all in Logstash. You first need to use grok or dissect to separate out the fields so you get the IP in a single field. Then you need to use the geoip plugin to add location information based on this IP. The location field, which contains a lat and a lon field, then need to be mapped as a geo_point field in an index template. This blog post may also be useful.