@corey.robertson Thanks for that! Was really useful.
I've followed the example but on the last step (reindex) I get the following error:
{
"index" : "logstash_new",
"type" : "_doc",
"id" : "CUBb5HUBAXq168Vsitkz",
"cause" : {
"type" : "mapper_parsing_exception",
"reason" : "failed to parse field [geoip_src_fixed.location] of type [geo_point]",
"caused_by" : {
"type" : "parse_exception",
"reason" : "[lat] and [lon] must be valid double values",
"caused_by" : {
"type" : "number_format_exception",
"reason" : "empty String"
}
}
},
"status" : 400
I'm not entirely sure what is exactly going wrong, but based on the error it seems like the mapping has gone wrong and the [geoip_src_fixed.location] is receiving an empty value. Any help to work out what I'm doing wrong would be appreciated.
To help, the original mapping of logstash includes
{
"logstash" : {
"mappings" : {
"properties" : {
"geoip_dst" : {
"properties" : {
"location" : {
"properties" : {
"lat" : {
"type" : "float"
},
"lon" : {
"type" : "float"
}
}
I therefore have created the following mapping
PUT /logstash_new
{
"mappings": {
"properties": {
"geoip_src_fixed" : {
"properties": {
"location" : {
"type" : "geo_point"
}
}
},
"geoip_dst_fixed" : {
"properties": {
"location" : {
"type" : "geo_point"
}
}
}
}
}
}
And to create the ingestion pipe I have then used
PUT /_ingest/pipeline/convert_geo
{
"processors": [
{
"set": {
"field": "geoip_src_fixed.location.lat",
"value": "{{geoip_src.location.lat}}"
}
},
{
"set": {
"field": "geoip_src_fixed.location.lon",
"value": "{{geoip_src.location.lon}}"
}
},
{
"set": {
"field": "geoip_dst_fixed.location.lat",
"value": "{{geoip_dst.location.lat}}"
}
},
{
"set": {
"field": "geoip_dst_fixed.location.lon",
"value": "{{geoip_dst.location.lon}}"
}
}
]
}
And finally run the following reindex command, which throws the above error
POST _reindex/
{
"source": {
"index": "logstash"
},
"dest": {
"pipeline": "convert_geo",
"index": "logstash_new"
}
}