Hi Guys,
So I have an issue with displaying the geolocation data on Kibana map visualization. Sorry if my description becomes too long but I will try to provide as much information as possible.
Past:
I am using logstash to ingest data, parse/transform, and send over to elastic search. I had logs coming in from WAF but everything within the message field. Converted that into JSON format to perform further operations. One of those fields was IP field. I activated geoip option for the filter plugin in logstash assuming I might be able to plot the country on a map based on source IP.
Present:
Logs are coming in as they should and I am able to create visualizations using other data points. The only thing I am not able to do is map the locations of source IP on kibana map. When I try to add a layer on kibana map using the 'Document' option, I do not see the desired index in the drop-down.
I did a bit of searching and found that in order for my index to be visible as a valid option I need to have a geo_point data type in my index. I looked at the available data fields in my index related to geo location and found multiple fields and figured out that I would need geoip.location field as geo-point data type. The issue is I have no idea as to how to do that. I looked at the mapping of the fields in index setting page and found that the fields are mapped to float (I have provided information below).
I also looked at logstash documentation for geoip option and it states that mapping is automatically done to geo_data point when using elasticsearch output plugin. But i don't see the same.
Logstash Setting:
filter {
json {
source => "message"
}
ruby {
path => "/etc/logstash/rubyScripts/flattenJSON.rb"
script_params => { "field" => "httpRequest" }
}
ruby {
path => "/etc/logstash/rubyScripts/headersFaltten.rb"
script_params => { "field" => "httpRequest.headers" }
}
geoip {
source => "httpRequest.clientIp"
}
date {
match => ["timestamp", "UNIX_MS"]
target => "@timestamp"
}
mutate {
remove_field => ["message", "timestamp", "httpRequest.country"]
}
}
GeoIP fields available in my index:
geoip.city_name, geoip.continent_code, geoip.country_code2, geoip.country_code3, geoip.country_name, geoip.dma_code, geoip.ip, geoip.latitude, geoip.location.lat, geoip.location.lon, geoip.longitude, geoip.postal_code, geoip.region_code, geoip.region_name, geoip.timezone
Default Geoip mapping setting in my index:
"geoip": {
"properties": {
"city_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dma_code": {
"type": "long"
},
"ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"latitude": {
"type": "float"
},
"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
},
"longitude": {
"type": "float"
},
"postal_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
From logstash documentation online:
"A [geoip][location] field is created if the GeoIP lookup returns latitude and longitude. The field is stored in the GeoJSON format. Additionally, the default Elasticsearch template provided with the elasticsearch output maps the [geoip][location] field to an Elasticsearch Geo_point datatype."
Please help me with this issue. And also let me know if you need any more information.
Kibana version: 7.7
Experience with ELK: 3-4 months
We aren't all guys though.