I can't seem to get the parsing of the GeoPoint data correct so that it can be used by Kibana's Tile Map
Here is a data sample...
2016-06-06 11:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4
2016-06-06 11:53:59,839 DEBUG longitude:11.621748 latitude:48.162687 count:4
2016-06-06 11:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4
2016-06-06 11:53:59,839 DEBUG longitude:11.621748 latitude:48.162687 count:4
2016-06-06 11:53:59,839 DEBUG longitude:-1.136306 latitude:53.950168 count:4
2016-06-06 11:53:59,839 DEBUG longitude:-3.71067 latitude:40.34683 count:4
2016-06-06 16:53:59,839 DEBUG longitude:6.118425 latitude:49.627499 count:4
2016-06-06 16:53:59,839 DEBUG longitude:136.921031 latitude:35.165557 count:12
2016-06-06 16:53:59,839 DEBUG longitude:23.706605 latitude:60.747242 count:4
2016-06-06 16:53:59,839 DEBUG longitude:-3.71067 latitude:40.34683 count:4
2016-06-06 16:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4
Logstash conf.
input {
file {
path => "path to data.log"
codec => multiline {
pattern => "[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}"
negate => true
what => previous
}
}
}
filter {
# Sorting out the data
grok {
match => { "message" => "(?<ts>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3})%{SPACE}(?<msg>.*)" }
}
# Correcting the timezone
date {
match => [ "ts" , "yyyy-MM-dd HH:mm:ss,SSS -02:00" ]
timezone => "UTC"
}
grok {
match => {"msg" => "%{LOGLEVEL:Loglevel}[ ]longitude:(?<Longitude>[0-9.-]+) latitude:(?<Latitude>[0-9.-]+) count:(?<GeoPointCount>[0-9]+)"}
}
# Only generates this info IF longitude exists. (this is just an extract)
if [Longitude]{
# First convert to a float
mutate {
convert => { "Latitude" => "float" }
convert => { "Longitude" => "float" }
mutate{
rename => {
"Latitude" => "[location][lat]"
"Longitude" => "[location][lon]"
}
}
}
#############
# Cleanup
#############
# Clean up any parsing errors
if "_grokparsefailure" in [tags]{
mutate {
remove_tag => [ "_grokparsefailure" ]
}
}
# Remove the unnecessary TempVariables
mutate {
remove_field => ["msg"]
}
}
output {
elasticsearch {
# protocol => "http"
hosts => "0.0.0.0:9200"
index => "geoPointData-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
Add in the elasticSearch geoPoint mapping
curl -XPUT 'http://localhost:9200/geoPointData*/_mapping/mapPin' -d '
{
"mapPin" : {
"properties" : {
"location" : {
"type" : "geo_point",
"lat_lon" : true,
"fielddata" : {
"format" : "compressed"
}
}
}
}
}'
Then once the data is parsed it can be seen in Kibana... (example json)
{
"_index": "geoPointData-2016.06.07",
"_type": "logs",
"_id": "AVUqGkF54r3c695U7Pnx",
"_score": null,
"_source": {
"path": "path to data.log",
"Loglevel": "DEBUG",
"@timestamp": "2016-06-07T09:03:30.967Z",
"GeoPointCount": "4",
"@version": "1",
"host": "xxx",
"location": {
"lon": 6.079736,
"lat": 51.131458
},
"message": "2016-06-06 16:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4",
"ts": "2016-06-06 16:53:59,839",
},
"fields": {
"@timestamp": [
1465290210967
]
},
"sort": [
1465290210967
]
}
Then when I got to the tile visualisation, the field 'location' can be found/seen by the Tile Map. But no matter how I play with the data, whenever I hit play I always see "No results found".
Clearly I am doing something wrong. Please can someone help me.