Convert lat and long to geohash

Elastic Stack Kibana
1

I have done quite a bit of searching, but either don't know what I am doing wrong, or don't know the right question to ask.

I am ingesting data, including latitude and longitude, and converting them to floats like this:

filter {
ruby
{
    path => "/opt/bitnami/logstash/chop.rb"
}
mutate
{
    rename =>
    {
        "stop_lon" => "[location][lon]"
        "stop_lat" => "[location][lat]"
    }
}

However, I need to convert the location to a geohash so I can do mapping, but I am struggling to figure out how to do that. 'Location' doesn't seem to be a field in my index, but is just two floats that I generate in chop.rb.

How do I generate this?

2

Perhaps see this thread. In short you need a index template with a geo_point data type mapping typically accomplished via an index_template

and perhaps this post

3

Maybe I am going about this wrong. I am trying to visualize data in a coordinate map, which requires a geohash type. I have the mutation I included above, and I can see that the location.lat and location.lon are numbers in my index (through the GUI).

You were suggesting that I needed to apply a template, but this is the log during startup.

[2019-09-02T15:28:53,962][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}

Does this mean that its already being applied? "location"=>{"type"=>"geo_point"}

4

hi @Brian_Seel

First What Version are you on?

2nd my understanding is that there is no data type geohash, geohash is an aggregation used to aggregate geo_point data.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-geohashgrid-aggregation.html

Apologies but from the one line log line I can not tell, not enough context.

Where did that log come from Logstash?
Did you create an index_template manually like in the thread referenced, are you using a beat module?
Are you trying to manage the template from Logstash?
Both?

The typically flow if you are creating custom data is ..

  1. Create and index template with the desired data types
  2. Create your logstash to read, transform and load your data
  3. Iterate 1 & 2 till you get it right, may include delete the index template until you get it right
  4. Discover and visualize the data. If the data has a geo_point data type it should show up automatically in the drop downs when you try to create a visualization.

But yes based on that context if "looks" like it is there is a geo_point data type...

you can check by running these commands in the Dev tools

GET /name-of-index/_mapping

And look at the mappings you should see something like this....

{
  "city_center" : {
    "mappings" : {
      "properties" : {
        "location" : {
          "type" : "geo_point"
        },
        "name" : {
          "type" : "text"
        }
      }
    }
  }
}

and if you run a search it should look something like

GET /city/_search

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "city_center",
        "_type" : "_doc",
        "_id" : "oAzB8mwBheMK44zyIkBj",
        "_score" : 1.0,
        "_source" : {
          "name" : "stamford",
          "location" : [
            -73.554212,
            41.053923
          ]
        }
      }
    ]
  }
}

You should also be able to see the index_templates using this

GET /_template/name-of-template

5

Thank you for the response. To anyone that has this issue in the future, the thing I was doing wrong was to try and create the template, and then try and apply it to the same index. You can't do that. You have to reindex into a new index.

That means making the template, like this post says.

PUT my_index 
{
  "mappings": {
    "doc": { 
      "properties": { 
        "location":    { "type": "geo_point"  }
      }
    }
  }
}

But then reindexing:

POST _reindex
{
  "source": {
    "index": "gtfs-data"
  },
  "dest": {
    "index": "my_index1"
  }
}

Thanks for the help, Stephen!

2 Likes
6

Cool

That's correct and to be a little bit more clear / directional.... Since In general, you can't change the field type once the index is created

either you
a) create the template with the specified field type before you index data or

B) like you showed create a new index with the correct template and re-index into it

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.