Kibana 4 and geoip


(Paul) #1

Hi Guys,
I have created some indexes on ES, using LogStash, and then trying to visualise with Kibana 4.

To visualise the geo data I try to select a 'Tile Map' and then 'Aggregation' with Geohash, and finally 'Field' of geoip.location.

This fails with :

Error: Request to Elasticsearch failed: {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[uyUOwArCTO2S1muPN1HAiw][logstash-2015.01][0]: ClassCastException[org.elasticsearch.index.fielddata.plain.DoubleArrayIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexGeoPointFieldData]}{[uyUOwArCTO2S1muPN1HAiw][logstash-2015.02][0]: 

Clearly something is wrong. Have I mucked up my index? I have set the geop field to be "not analyzed" and I have left the geop.location fields as is. Any clues what is wrong?

Here is a sample of the geoip.location field:

geoip.location	 	["4.900000000000006","52.36670000000001"]

And here is the mapping:

          "geoip": {
        "properties": {
          "timezone": {
            "type": "string"
          },
          "region_name": {
            "type": "string"
          },
          "real_region_name": {
            "type": "string"
          },
          "postal_code": {
            "type": "string"
          },
          "longitude": {
            "type": "double"
          },
          "location": {
            "type": "double"
          },
          "latitude": {
            "type": "double"
          },
          "area_code": {
            "type": "long"
          },
          "city_name": {
            "type": "string"
          },
          "continent_code": {
            "type": "string"
          },
          "country_code2": {
            "type": "string"
          },
          "country_code3": {
            "type": "string"
          },
          "country_name": {
            "type": "string"
          },
          "dma_code": {
            "type": "long"
          },
          "ip": {
            "type": "string"
          }
        }
      },

JSON latitude/longitude to geo_point
(Mark Walkom) #2

It needs to be "type": "geo_point" for geoip.location.


(Paul) #3

Thanks you Mark. I feared this was the case.

What is the quickest way of me correcting this? How do I change the type for a field? Do I have to index?? oh no! Or can I do that efficiently?

Update: Reading some posts and docs implies I might be able to do this by greating a new index with the correct mapping, and then doing a scrolled search to pull in the data from the old index?

Can someone expand on the mechanics and practicality of doing this? I have 5 indexes and they each have about 80 million docs so the re-indexing is not trivial.


(Mark Walkom) #4

Just use this to reindex with Logstash - https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06


(Paul) #5

Thanks again Mark. I will test it out.
Could you outline what it is doing? Looks like it is doing the scrolled search and then outputing back to the new index?
I presume I should delete and add a new corrected template before doing this, so that I have the correct mapping for the geoip.location field?
Ta
Paul


(Mark Walkom) #6

That's what it does, yes.

You will need to update the template, you will also need to reindex to a new index name, you can't just do it over the top of the existing one.


(Paul) #7

Where do I specify the name of the new index? The logstash conf example you used some metadata but do I set it in the {output} section in the usual way?
eg;
index => "new_log-%{+YYYY.MM}"

Thanks


(Paul) #8

Ok, I have the re-index working. My new index has a lovely geoip.location field that is type: geo_point.

              "location" : {
            "type" : "geo_point"
          },

And the actual data looks ok as well:

geoip.location	 	[-73.627,4.142]

However when I go over to Kibana and try to visualise on a Tile Map I get an error. I select "geohash" for the aggregation (the only option) and there is no option available for "field". Any ideas? Why no geoip.location available?


(Paul) #9

Fixed it : Go to "Settings" on the main Kibana dashboard. Then delete the old index pattern and add the new index pattern. Check the geoip.location field is showing as "geo_point".

Without this refresh, Kibana was using the old type field for geoip.location


(Mark Walkom) #10

Yep, you can do this by issuing a refresh under the settings for the index.


(Paul) #11

Thanks Mark. Its all working.

One last question (promise!): Performance

What is the best way to do this for an ES cluster of 4 nodes, with 4 indexes, each index 4 shards (one per node)?

Should I do one index all local (input & output same node) or input on node1 output to node2?


(Mark Walkom) #12

One shard per node is good, don't forget replicas though! :smile:
However you are better off having an uneven number of nodes to ensure quorum.

You are also definitely better off spreading the indexing load across mutiple servers.


(Craig Tataryn) #13

I've tried doing a refresh, but my geoip.location is still listed as of type Number and don't have the option to change it to "geo_point" through the GUI.

As far as I know this was working a few days ago (I'm using the Cloud instance of Kibana). When I check my documents I do see proper data like so:

"geoip": {
  "ip": "XX.XX.XXX.XX",
  "location": [
    -111.8906,
    33.61189999999999
  ]
}

Is there something else I'm missing?


(Mark Walkom) #14

Check the mapping of the field, that'll tell you definitively.


(Craig Tataryn) #15

@warkolm Where do I find the mapping of the field?


(Mark Walkom) #16

Look at GET /indexname/_mapping


(bob.webman) #17

Like @warkolm said, GET /indexname/_mapping and here is what I get when I do that:
[Note the location <-> geo_point mapping]

          "geoip": {
            "properties": {
              "timezone": {
                "type": "string"
              },
              "region_name": {
                "type": "string"
              },
              "real_region_name": {
                "type": "string"
              },
              "postal_code": {
                "type": "string"
              },
              "longitude": {
                "type": "float"
              },
              "location": {
                "type": "geo_point"
              },
              "latitude": {
                "type": "float"
              },
              "ip": {
                "type": "string"
              },
              "area_code": {
                "type": "string"
              },
              "city_name": {
                "type": "string"
              },
              "continent_code": {
                "type": "string"
              },
              "coordinates": {
                "type": "double"
              },
              "country_code2": {
                "type": "string"
              },
              "country_code3": {
                "type": "string"
              },
              "country_name": {
                "type": "string"
              },
              "dma_code": {
                "type": "string"
              }
            },

(Magnus Bäck) #18

Perhaps you just need to refresh the field list in Kibana?


(Craig Tataryn) #19

Where as mine is:

      "geoip": {
        "properties": {
          "area_code": {
            "type": "long"
          },
          "city_name": {
            "type": "string"
          },
          "continent_code": {
            "type": "string"
          },
          "country_code2": {
            "type": "string"
          },
          "country_code3": {
            "type": "string"
          },
          "country_name": {
            "type": "string"
          },
          "dma_code": {
            "type": "long"
          },
          "ip": {
            "type": "string"
          },
          "latitude": {
            "type": "double"
          },
          "location": {
            "type": "double"
          },
          "longitude": {
            "type": "double"
          },
          "postal_code": {
            "type": "string"
          },
          "real_region_name": {
            "type": "string"
          },
          "region_name": {
            "type": "string"
          },
          "timezone": {
            "type": "string"
          }
        }
      },

When I went to change the type (see previous post) it didn't give me the option to choose geo_point.

Craig.


(Craig Tataryn) #20

I have tried that but it doesn't seem to help. I go into Kibana, click on Settings then I click on the index pattern and finally click the yellow Refresh icon.

Craig.