GeoIP data present but cannot be displayed

[rcoundon]

I'm not sure if this is a Kibana issue or an Elasticsearch issue.

I have a logstash pipeline that is submitted data from an nginx access.log to ES. I'm using the geoip plugin to resolve the clientip to an address and lat/long. This all works and I see the following data in Kibana:

# geoip.latitude		51.347
# geoip.location.lat		51.347
# geoip.location.lon		-0.787
# geoip.longitude		-0.787
t geoip.postal_code		GU47
t geoip.region_code		BRC
t geoip.region_name		Bracknell Forest

However, if I try to create a visualisation for this I get the message and select geohash as the aggregation I get the message:
No Compatible Fields: The "test*" index pattern does not contain any of the following field types: geo_point

In the index definition I have:

"geoip": {
"properties": {
    "city_name": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "continent_code": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "country_code2": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "country_code3": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "country_name": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "ip": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "latitude": {
        "type": "float"
    },
    "location": {
        "properties": {
            "lat": {
                "type": "float"
            },
            "lon": {
                "type": "float"
            }
        }
    },
    "longitude": {
        "type": "float"
    },
    "postal_code": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "region_code": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "region_name": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    },
    "timezone": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword",
                "ignore_above": 256
            }
        }
    }
}
}

Can anyone help me see what's going wrong here?

Many thanks

[Bill_McConaghy]

The issue lies in the mapping for the index. See this documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html for details. TLDR, your location attribute should look like this:

"location": { "type": "geo_point" }

[rcoundon]

Thanks Bill, I had taken a look at that but I think what's gotten me a bit confused is what I'd read on this blog: https://www.elastic.co/blog/geoip-in-the-elastic-stack
Given I'm using the geoip plugin as described in that blog and the index is automatically created via the elasticsearch output plugin, why isn't the location attribute it being created as a geo_point type automatically? Is there something additional I need to do to make that happen?

[Bill_McConaghy]

I think the issue is that the automatic mapper is not smart enough to see that that is a geoip point. So you will need to recreate the index with the right mapping for that field and reindex the data.

[rcoundon]

I've gone and deleted the old index and had logstash recreate it and I can now create a visualisation referencing the location.
This would seem to imply that the types are correct now but something else is going on. I definitely have data resolved to IP addresses in the index but the problem I have now is that no data is shown. Where to look now?

[Bill_McConaghy]

Not sure what you mean when you say no data is shown. In Discover?

[rcoundon]

Sorry, fair point, I wasn't clear. I see the data in Discover with lat and long:

But not in my visualization which is defined like:

[rcoundon]

Ok, I think I've gotten this working
I defined an entirely new index specifying the location attribute as a geo_point as you first described.
I then used _reindex to copy the documents from the old to the new, and repointed my logstash servers at the new index and the data is showing on the map.
I guess I just found it a little odd that the visualization seemed to think the data type was fine in the definition of that visualization but couldn't actually display it.
Thanks very much for taking the time to help me

[Bill_McConaghy]

Awesome, glad things are working for you now. I think the setup could be improved for sure. It might be worth putting in a logstash Github issue outlining the steps you went through and the pain points. I know they are working on streamlining things and stories like this could be helpful in that effort.

[rcoundon]

Thanks Bill, no problem, I'll do that