Post ingest, how can I generate lat long fields into geo point

I've ingested CSV data via Apache NiFi and posted to Elasticsearch. I've come to realize that there are "lat" and "lon" values in my index. My index is already in the 10's of millions of records.

`{"name":"Sam", "lat": 11.2345, "lon": 54.235634}`

How would I convert this into an acceptable geopoint format so that I can map it in Kibana?

Would I have to add another field that incorporates both lat/lon? Would I have to delete it all and try to append an acceptable geopoint (I'd rather not)? Is there way to create a geopoint field by combining known fields? Can I create a geopoint field post ingest?

I understand how mapping fields works, but if I'm ingesting data that I don't know what's fully in it, I would like to know what post ingest options I have.

Thank you all.

1 Like

Maybe something like this after you change mapping for location to geo-point

POST your_index/_update_by_query
{
  "script": {
    "inline": "ctx._source.location = ctx._source.lat, ctx._source.lon",
    "lang": "painless"
  },
  "query": {
    "match_all": {}
  }
}

You are just creating a new field called location with the lat/long combined in a string. Which is the 2nd example on https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

Thank you. After a little fixing around, here is what I came up with.

1. After initial ingest, I create mapping of soon to be created "coordinates" field

PUT /my-index/_mapping
{
  "properties": {
    "coordinates": {
      "type": "geo_point"
    }
  }
}

2. Run script to generate new "coordinates" field

POST my-index/_update_by_query
{
  "script": {
    "inline": "ctx._source.coordinates = [ctx._source.lon, ctx._source.lat]",
    "lang": "painless"
  },
  "query": {
    "match_all": {}
  }
}

I changed the inline field to be in brackets because I kept getting errors of the "," between the lon and lat fields. Also I changed the order to lon,lat because that's how in the documentation it said to order it

3. Open up Maps and find your index available to start mapping!

2 Likes

Any suggestion you have for nested values? Here is another mapping of a field.

I cannot do ctx._source.map.position.latitude in the "inline" section.

"map" : {
          "properties" : {
            "position" : {
              "properties" : {
                "latitude" : {
                  "type" : "float"
                },
                "longitude" : {
                  "type" : "float"
                }
              }
            }
          }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.