I'm having an issue getting Kibana to generate a coordinate map using a geo_point variable that I pass in from Logstash.
I'm parsing bro logs from a syslog file. Within the filter section of my conf file, I have the following code:
if [json][id.resp_h] !~ /^10\./
{
{
geoip {
source => "[json][id.resp_h]"
target => "[json][resp_geoip]"
}
}
}
To be clear, I parsed the lines of the syslog file (in the case from a Bro conn log) into an array I called 'json'. Then the array has a variety of elements, including one called 'id.resp_h', which is an IP address. I then use the geoip filter to generate a new array element called resp_geoip.
The output section of my conf file is as follows:
output
{
elasticsearch {
action => "index"
hosts => ["localhost:9200"]
index => "logstash-bro_syslog-%{+YYYY.MM.dd}"
manage_template => true
template => "C:\Bitnami\elk-6.5.4-0\logstash\bro_template.json"
template_overwrite => true
document_type => "bro_logs"
}
stdout {
}
}
As shown, I'm using a template called bro_template.json. Within that file, I specify the type as follows:
"resp_geoip" : {
"dynamic" : "true",
"properties" : {
"timezone" : {
"type" : "text"
},
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
"continent_code" : {
"type" : "text"
},
"city_name" : {
"type" : "text"
},
"country_name" : {
"type" : "text"
},
"dma_code" : {
"type" : "integer"
},
"country_code2" : {
"type" : "text"
},
"country_code3" : {
"type" : "text"
},
"region_name" : {
"type" : "text"
},
"postal_code" : {
"type" : "text"
}
}
},
With the above configuration, all my data appears in Kibana and looks to be formatted correctly. That being the case, the variable that I want to map in Kibana shows up as 'resp_geoip.location'.
The problem is I cannot get Kibana to display anything on the coordinate map. I select Coordinate Map on the Visualization page, then select the index with which I'm working, and then get to the map setup page. Under Metrics, I select 'Count'. Under buckets, I click 'Geo Coordinates', which then asks for an Aggregation. The only option shown is Geohash (which I don't have in my data set). But under 'select field', Kibana does offer me the option of choosing my variable above - resp_geoip.location. But when I select it and then push the button to generate the map, nothing appears.
So, at this point I'm at a loss. I defined a template to cover the geo_point conversion as I've seen others suggest, and I've also confirmed that logstash's geoip filter is working as expected. But nothing I seem to do will generate the map.
Am I supposed to be creating a geohash to be used in that map? If so, is there a function available to do so? I've checked around and haven't seen anything ELK-specific. Or is the functionality I'm describing above as expected and I just have some other weird anomaly going on?
I'll note that not every data record has a resp_geoip.location value. Some are indeed blank. I've tried doing a search in the search bar to narrow down only the ones that have values, but that still hasn't solved the issue.
Thanks for your assistance in solving this frustrating issue!