Geoip filter in logstash with Index Template not creating geo_point object as expected

[ehfdub]

Hello,

I have a geoip in my logstash filter:

geoip { source => "src_ip" }

and I created an index template via the Kibana DevTools for ALL new indexes:

PUT /_template/my_template
{
  "order": 0,
  "template": "*",  
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

which appears as expected when I query it from Elasticsearch:

{
  "my_template" : {
    "order" : 0,
    "index_patterns" : [
      "*"
    ],
    "settings" : { },
    "mappings" : {
      "properties" : {
        "location" : {
          "type" : "geo_point"
        }
      }
    },
    "aliases" : { }
  }
}

but when a new index is created, the location.lat and location.lon fields still get mapped to floating point values instead of a geo_point.

Can anyone point me to what I'm missing (not understanding) here?

Thanks,
Ehf

[Dzmitry]

Hi @ehfdub

I followed the recents docs and successfully created an index with geopoint field in Kibana 7.15.1, that was later recognized during index pattern creation in Kibana:

And this how it looks on Discover

What stack version do you use? Can you give it a try manually without logstash? If it will work, then you need to change logstash configuration, you can check the issue example here

Regards, Dzmitry

[leandrojmp (Leandro Pereira)]

Your issue here is that the field you mapped as geo_point and the field with the location are not the same.

You mapped the location field, but your geolocation field is named geoip.location.

You should change your mapping or use target => "location" in your geoip filter on logstash.

[ehfdub]

Thanks @leandrojmp

I appreciate your keen eye. Per your suggestion, I just updated my index template to

{
  "my_template" : {
    "order" : 0,
    "index_patterns" : [
      "*"
    ],
    "settings" : { },
    "mappings" : {
      "properties" : {
        "geoip.location" : {
          "type" : "geo_point"
        }
      }
    },
    "aliases" : { }
  }
}

I will check back tomorrow after a new index is created and verify it's working as expected.

Thanks,
ehf

[leandrojmp (Leandro Pereira)]

It won't work with this mapping, it is wrong.

It should be:

"mappings": {
    "properties" : {
        "geo": {
            "properties": {
                "location": {
                    "type": "geo_point"
                }
            }
        }
    }
}

You have a json object named geoip with a field named location, kibana will show this as geoip.location.

Using geoip.location in your mapping means that you have a field named geoip.location, where the . is a literal dot, not a json object flattened.

[ehfdub]

Thanks @leandrojmp.

Is there anything I need to do other than refresh the index in Kibana to get the geo_point type to show up correctly? It still shows up as two separate float fields rather than a `geo_point.'

I created the template using Kibana devtools:

{
  "my_template" : {
    "order" : 0,
    "index_patterns" : [
      "*"
    ],
    "settings" : { },
    "mappings" : {
      "properties" : {
        "geo" : {
          "properties" : {
            "location" : {
              "type" : "geo_point"
            }
          }
        }
      }
    },
    "aliases" : { }
  }
}

I waited for a new index to be created and verified that the geo_point mapping was created correctly in that new index under the geoip.location field.

 "geoip": {
          "properties": {
            "city_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "continent_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_code2": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_code3": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "dma_code": {
              "type": "long"
            },
            "ip": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            },
            "postal_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "timezone": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },

But viewing Kibana Analytics | Discover still shows the float fields

One thought: Is it okay to keep older indice which were created before this new index mapping was created? (do those older indice have to be deleted?)

Thanks,
Ehf