Kibana 4 and geoip

elastic_paul · June 1, 2015, 9:26pm

Hi Guys,
I have created some indexes on ES, using LogStash, and then trying to visualise with Kibana 4.

To visualise the geo data I try to select a 'Tile Map' and then 'Aggregation' with Geohash, and finally 'Field' of geoip.location.

This fails with :

Error: Request to Elasticsearch failed: {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[uyUOwArCTO2S1muPN1HAiw][logstash-2015.01][0]: ClassCastException[org.elasticsearch.index.fielddata.plain.DoubleArrayIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexGeoPointFieldData]}{[uyUOwArCTO2S1muPN1HAiw][logstash-2015.02][0]:

Clearly something is wrong. Have I mucked up my index? I have set the geop field to be "not analyzed" and I have left the geop.location fields as is. Any clues what is wrong?

Here is a sample of the geoip.location field:

geoip.location	 	["4.900000000000006","52.36670000000001"]

And here is the mapping:

          "geoip": {
        "properties": {
          "timezone": {
            "type": "string"
          },
          "region_name": {
            "type": "string"
          },
          "real_region_name": {
            "type": "string"
          },
          "postal_code": {
            "type": "string"
          },
          "longitude": {
            "type": "double"
          },
          "location": {
            "type": "double"
          },
          "latitude": {
            "type": "double"
          },
          "area_code": {
            "type": "long"
          },
          "city_name": {
            "type": "string"
          },
          "continent_code": {
            "type": "string"
          },
          "country_code2": {
            "type": "string"
          },
          "country_code3": {
            "type": "string"
          },
          "country_name": {
            "type": "string"
          },
          "dma_code": {
            "type": "long"
          },
          "ip": {
            "type": "string"
          }
        }
      },

warkolm · June 2, 2015, 11:19am

It needs to be "type": "geo_point" for geoip.location.

elastic_paul · June 2, 2015, 4:00pm

Thanks you Mark. I feared this was the case.

What is the quickest way of me correcting this? How do I change the type for a field? Do I have to index?? oh no! Or can I do that efficiently?

Update: Reading some posts and docs implies I might be able to do this by greating a new index with the correct mapping, and then doing a scrolled search to pull in the data from the old index?
https://www.elastic.co/blog/changing-mapping-with-zero-downtime

Can someone expand on the mechanics and practicality of doing this? I have 5 indexes and they each have about 80 million docs so the re-indexing is not trivial.

warkolm · June 2, 2015, 9:43pm

Just use this to reindex with Logstash - https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06

elastic_paul · June 2, 2015, 10:19pm

Thanks again Mark. I will test it out.
Could you outline what it is doing? Looks like it is doing the scrolled search and then outputing back to the new index?
I presume I should delete and add a new corrected template before doing this, so that I have the correct mapping for the geoip.location field?
Ta
Paul

warkolm · June 2, 2015, 11:06pm

That's what it does, yes.

You will need to update the template, you will also need to reindex to a new index name, you can't just do it over the top of the existing one.

elastic_paul · June 3, 2015, 6:52am

Where do I specify the name of the new index? The logstash conf example you used some metadata but do I set it in the {output} section in the usual way?
eg;
index => "new_log-%{+YYYY.MM}"

Thanks

elastic_paul · June 3, 2015, 10:30am

Ok, I have the re-index working. My new index has a lovely geoip.location field that is type: geo_point.

              "location" : {
            "type" : "geo_point"
          },

And the actual data looks ok as well:

geoip.location	 	[-73.627,4.142]

However when I go over to Kibana and try to visualise on a Tile Map I get an error. I select "geohash" for the aggregation (the only option) and there is no option available for "field". Any ideas? Why no geoip.location available?

elastic_paul · June 3, 2015, 11:26am

Fixed it : Go to "Settings" on the main Kibana dashboard. Then delete the old index pattern and add the new index pattern. Check the geoip.location field is showing as "geo_point".

Without this refresh, Kibana was using the old type field for geoip.location

warkolm · June 3, 2015, 12:01pm

Yep, you can do this by issuing a refresh under the settings for the index.

elastic_paul · June 3, 2015, 12:38pm

Thanks Mark. Its all working.

One last question (promise!): Performance

What is the best way to do this for an ES cluster of 4 nodes, with 4 indexes, each index 4 shards (one per node)?

Should I do one index all local (input & output same node) or input on node1 output to node2?

warkolm · June 3, 2015, 11:13pm

One shard per node is good, don't forget replicas though!
However you are better off having an uneven number of nodes to ensure quorum.

You are also definitely better off spreading the indexing load across mutiple servers.

ctataryn · July 13, 2016, 9:45pm

I've tried doing a refresh, but my geoip.location is still listed as of type Number and don't have the option to change it to "geo_point" through the GUI.

As far as I know this was working a few days ago (I'm using the Cloud instance of Kibana). When I check my documents I do see proper data like so:

"geoip": {
  "ip": "XX.XX.XXX.XX",
  "location": [
    -111.8906,
    33.61189999999999
  ]
}

Is there something else I'm missing?

warkolm · July 13, 2016, 10:22pm

Check the mapping of the field, that'll tell you definitively.

ctataryn · July 14, 2016, 12:52am

@warkolm Where do I find the mapping of the field?

warkolm · July 14, 2016, 12:53am

Look at GET /indexname/_mapping

bob_webman · July 14, 2016, 4:55am

Like @warkolm said, GET /indexname/_mapping and here is what I get when I do that:
[Note the location <-> geo_point mapping]

          "geoip": {
            "properties": {
              "timezone": {
                "type": "string"
              },
              "region_name": {
                "type": "string"
              },
              "real_region_name": {
                "type": "string"
              },
              "postal_code": {
                "type": "string"
              },
              "longitude": {
                "type": "float"
              },
              "location": {
                "type": "geo_point"
              },
              "latitude": {
                "type": "float"
              },
              "ip": {
                "type": "string"
              },
              "area_code": {
                "type": "string"
              },
              "city_name": {
                "type": "string"
              },
              "continent_code": {
                "type": "string"
              },
              "coordinates": {
                "type": "double"
              },
              "country_code2": {
                "type": "string"
              },
              "country_code3": {
                "type": "string"
              },
              "country_name": {
                "type": "string"
              },
              "dma_code": {
                "type": "string"
              }
            },

magnusbaeck · July 14, 2016, 6:04am

Perhaps you just need to refresh the field list in Kibana?

ctataryn · July 14, 2016, 3:20pm

Where as mine is:

      "geoip": {
        "properties": {
          "area_code": {
            "type": "long"
          },
          "city_name": {
            "type": "string"
          },
          "continent_code": {
            "type": "string"
          },
          "country_code2": {
            "type": "string"
          },
          "country_code3": {
            "type": "string"
          },
          "country_name": {
            "type": "string"
          },
          "dma_code": {
            "type": "long"
          },
          "ip": {
            "type": "string"
          },
          "latitude": {
            "type": "double"
          },
          "location": {
            "type": "double"
          },
          "longitude": {
            "type": "double"
          },
          "postal_code": {
            "type": "string"
          },
          "real_region_name": {
            "type": "string"
          },
          "region_name": {
            "type": "string"
          },
          "timezone": {
            "type": "string"
          }
        }
      },

When I went to change the type (see previous post) it didn't give me the option to choose geo_point.

Craig.

ctataryn · July 14, 2016, 3:22pm

I have tried that but it doesn't seem to help. I go into Kibana, click on Settings then I click on the index pattern and finally click the yellow Refresh icon.

Craig.

Topic		Replies	Views
GeoIP data present but cannot be displayed Kibana	10	1631	May 16, 2018
Missing Geo_point field in geohash aggregate -! Kibana 4.5 Elasticsearch	8	3730	January 10, 2017
Elastic Tile Service - how tos Kibana	5	1073	August 16, 2016
Kibana4: geopoint visualization does not work Kibana	11	4052	July 6, 2017
Error when trying to add documents layer to maps Kibana maps	6	577	September 16, 2020

Kibana 4 and geoip

Related topics