Hoping someone can assist with where I'm going wrong with adding two additional geoip/geo_point fields. I've updated each of the default templates on Ubuntu to include the additional points, duplicating the pre-existing geoip config...
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/elasticsearch-template-es6x.json
{
"template": "logstash-*",
"version": 60001,
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"type": "text",
"norms": false
}
}
},
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"location": {
"type": "geo_point"
},
"latitude": {
"type": "half_float"
},
"longitude": {
"type": "half_float"
}
}
},
"src_geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"location": {
"type": "geo_point"
},
"latitude": {
"type": "half_float"
},
"longitude": {
"type": "half_float"
}
}
},
"dest_geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"location": {
"type": "geo_point"
},
"latitude": {
"type": "half_float"
},
"longitude": {
"type": "half_float"
}
}
}
}
}
}
}
But when using the logstash geoip plugin to populate them...
geoip {
source => "src_ip"
target => "src_geoip"
fields => [ "city_name", "continent_code", "country_code2", "country_name", "latitude", "longitude", "location", "postal_code", "region_name", "timezone" ]
database => "/usr/share/GeoIP/GeoLite2-City.mmdb"
tag_on_failure => "_src_geoip_failure"
}
I get two separate properties rather than a geo_point...
src_geoip.location.lat: xx.xxx
src_geoip.location.lon : xx.xxx
Looking at my indexes default mappings:
/logstash-*/_mapping/_default_
...it doesn't appear that the properties I added came over.
{
"logstash-2018.12.26.23" : {
"mappings" : {
"_default_" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"norms" : false,
"type" : "text"
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
},
"norms" : false,
"type" : "text"
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"geoip" : {
"dynamic" : "true",
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
}
}
}
}
}
}
It also feels weird poking at those deeply nested config files just to add a geo_point, but that's been indicated in forum posts and I can't find specific documentation on such.
- https://www.elastic.co/blog/geoip-in-the-elastic-stack : talks about the default template but fails to provide details relevant to updating it
- https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping : provides template update details, but not for the default templates; I'm confused how I can define a template by upload like that. since I need to have an index to upload into, and when I have an index it will already have loaded data with the wrong template.