I have been tweaking my mappings and logstash pipeline configurations for days now trying to figure out hwo to plot GeoIPs to no avail. I have no idea what to do anymore.
I have a template:
{
"template" : "httpd*",
"version" : 1,
"settings" : {
"index.refresh_interval" : "5s",
"number_of_replicas": 0
},
"mappings": {
"apache_logs": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"geoip": {
"properties": {
...
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
},
...
}
},
... # cut out for char limit
}
}
}
}
I use this logstash configuration:
input {
beats {
port => 5044
tags => "apache"
}
}
filter {
if "apache" in [tags] {
grok {
match => { "message" => ["%{COMBINEDAPACHELOG}", ........ ] }
remove_field => "message"
tag_on_failure => ["no_apache_match"]
}
if "no_apache_match" not in [tags] {
geoip {
source => "clientip"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
}
mutate {
convert => {
"[geoip][coordinates]" => "float"
}
}
...
}
}
output {
if "apache" in [tags] {
if "apache_fiesta" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
user => "elastic"
password => "MrR0bot$"
index => "httpd_fiesta-%{+YYYY.MM.dd}"
document_type => "apache_logs"
}
}
}
...
}
I check that the mapping is being applied:
# curl -u elastic GET 'http://localhost:9200/httpd_fiesta-2017.10.09/_mapping'
{
"httpd_fiesta-2017.10.09": {
"mappings": {
"apache_logs": {
"properties": {
.....
"geoip": {
"properties": {
....
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
},
...
}
},
...
}
}
}
}
}
So geoip.location is indeed a geo_point data type.
In Kibana, I have refreshed the field list for my index pattern.
I have deleted this index prior to any recent changes, so all of my log entries are uniform.
In the Discover page, I have documents that look like this:
{
"_index": "httpd_fiesta-2017.10.09",
"_type": "apache_logs",
"_source": {
....
"geoip": {
...
"location": {
"lon": -106.7177,
"lat": 35.3275
},
...
},
...
"tags": [
"apache_fiesta",
"apache",
"beats_input_codec_plain_applied"
],
...
},
....
}
As you can see, there are no geoip failures, and the data is collected correctly.
Kibana shows a globe symbol next to geoip.location, and when i click it, it gives me the option to visualize.
On the visualize tab, I can select Aggregation->Geohash and Field->geoip.location just fine.
However, there is no data! I am using a valid time frame for which there is data in the discover tab.
The centroid label in the bottom right corner says NaN- NaN, which tells me it is not aggregating anything.
What the heck is going on?