[SOLVED] Parsing GeoPoint - Tile map shows - No results found

Elastic Stack Kibana
1

I can't seem to get the parsing of the GeoPoint data correct so that it can be used by Kibana's Tile Map

Here is a data sample...

2016-06-06 11:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4
2016-06-06 11:53:59,839 DEBUG longitude:11.621748 latitude:48.162687 count:4
2016-06-06 11:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4
2016-06-06 11:53:59,839 DEBUG longitude:11.621748 latitude:48.162687 count:4
2016-06-06 11:53:59,839 DEBUG longitude:-1.136306 latitude:53.950168 count:4
2016-06-06 11:53:59,839 DEBUG longitude:-3.71067 latitude:40.34683 count:4
2016-06-06 16:53:59,839 DEBUG longitude:6.118425 latitude:49.627499 count:4
2016-06-06 16:53:59,839 DEBUG longitude:136.921031 latitude:35.165557 count:12
2016-06-06 16:53:59,839 DEBUG longitude:23.706605 latitude:60.747242 count:4
2016-06-06 16:53:59,839 DEBUG longitude:-3.71067 latitude:40.34683 count:4
2016-06-06 16:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4

Logstash conf.
input {
file {
path => "path to data.log"

            codec => multiline {
                    pattern => "[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}"
                    negate => true
                    what => previous
            }
    }

}

filter {

    # Sorting out the data
    grok {
            match => { "message" => "(?<ts>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3})%{SPACE}(?<msg>.*)" }
    }

    
    # Correcting the timezone
    date {
            match => [ "ts" , "yyyy-MM-dd HH:mm:ss,SSS -02:00" ]
            timezone => "UTC"
    }


    grok {
             match => {"msg" => "%{LOGLEVEL:Loglevel}[ ]longitude:(?<Longitude>[0-9.-]+) latitude:(?<Latitude>[0-9.-]+) count:(?<GeoPointCount>[0-9]+)"}
    }



    # Only generates this info IF longitude exists. (this is just an extract)
    if [Longitude]{
            # First convert to a float
            mutate {
                    convert => { "Latitude" => "float" }
                    convert => { "Longitude" => "float" }
            mutate{
                    rename => {
                            "Latitude" => "[location][lat]"
                            "Longitude" => "[location][lon]"
                    }
            }
    }


    #############
    # Cleanup
    #############
    # Clean up any parsing errors
    if "_grokparsefailure" in [tags]{
            mutate {
                    remove_tag => [ "_grokparsefailure" ]
            }
    }
    # Remove the unnecessary TempVariables
    mutate {
            remove_field => ["msg"]
    }

}

output {
elasticsearch {
# protocol => "http"
hosts => "0.0.0.0:9200"
index => "geoPointData-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}

Add in the elasticSearch geoPoint mapping

curl -XPUT 'http://localhost:9200/geoPointData*/_mapping/mapPin' -d '
{
"mapPin" : {
"properties" : {
"location" : {
"type" : "geo_point",
"lat_lon" : true,
"fielddata" : {
"format" : "compressed"
}
}
}
}
}'

Then once the data is parsed it can be seen in Kibana... (example json)

{
"_index": "geoPointData-2016.06.07",
"_type": "logs",
"_id": "AVUqGkF54r3c695U7Pnx",
"_score": null,
"_source": {
"path": "path to data.log",
"Loglevel": "DEBUG",
"@timestamp": "2016-06-07T09:03:30.967Z",
"GeoPointCount": "4",
"@version": "1",
"host": "xxx",
"location": {
"lon": 6.079736,
"lat": 51.131458
},
"message": "2016-06-06 16:53:59,839 DEBUG longitude:6.079736 latitude:51.131458 count:4",
"ts": "2016-06-06 16:53:59,839",
},
"fields": {
"@timestamp": [
1465290210967
]
},
"sort": [
1465290210967
]
}

Then when I got to the tile visualisation, the field 'location' can be found/seen by the Tile Map. But no matter how I play with the data, whenever I hit play I always see "No results found".

Clearly I am doing something wrong. Please can someone help me.

2

Okay the issue is resolved. All about the mapping of course.

this is the correct mapping

curl -XPUT 'http://localhost:9200/lr*/_mapping/logs' -d '
{
"logs" : {
"properties" : {
"location" : {
"type" : "geo_point",
"lat_lon" : true,
"fielddata" : {
"format" : "compressed"
}
}
}
}
}'

(of course)