Set geopoint field from string field correctly for map visualization

Elastic Stack Elasticsearch
1

I have the following field in my documents:
"COORDINATES" : "-11.661306;-47.506833"

I have created a update_by_query that takes this field, extracts the lat/lon and sets to my actual geopoint defined field 'GeoLocation'.

The query im using for this is the following:

POST eventanalytics-*/_update_by_query
{
  "query": {
    "match_all": {}
  },
  "script": "def m = /(.*)\\;(.*)/.matcher(ctx._source.COORDINATES); if ( m.matches() ) { ctx._source.GeoLocation = [ m.group(2), m.group(1) ]; }"
}

This results in the GeoLocation field looking like this:

"GeoLocation" : [
            "-46.506833",
            "-23.661306"
          ]

When i go into Kibana Maps - add a layer.
Select Document
Select my index pattern
And it auto populates the Geospatial field section with 'GeoLocation'
I select 'Add Layer'
And it shows the layer under the layers tool bar. However, no items display on the screen. All documents in the index pattern have the new field populated. On scroll over of the layer name, it says it sees 10 documents.

Is my definition of the GeoLocation value incorrect from my script?

When i view the layer config, under 'Layer Style' it has 'Polygons' selection/highlighted. My assumption is that it should have 'Point' highlighted - as this is what it is. Do i need to have a update to the GeoLocation format to include: "type":"point"? If so, how can i update my script/query to get that format in place?

Could anyone lend any guidance?

2

Your field mapping needs to be a geo_point.

      "GeoLocation": {
        "type": "geo_point"
      }

Also your data needs to be numbers and not a string. So need to drop the "'s.

"GeoLocation" : [
            -46.506833,
            -23.661306
          ]
1 Like
3

I checked my mappings and i have the following:
"GeoLocation" : {
"type" : "geo_shape",
"ignore_malformed" : true
}

So yeah, its a shape, sigh. Can i force a change of the mapping to be geo_point?

I also updated my query to do the following:

POST eventanalytics-*/_update_by_query
{
  "query": {
    "match_all": {}
  },
  "script": "def m = /(.*)\\;(.*)/.matcher(ctx._source.COORDINATES); if ( m.matches() ) { ctx._source['GeoLocation.type'] = 'geo_point'; ctx._source['GeoLocation.coordinates'] = [ m.group(2), m.group(1) ]; }"
}

And i got the following message(s):

{
      "index": "eventanalytics-2021.11.05",
      "type": "_doc",
      "id": "retR8HwBxv25fxwK2IJ9",
      "cause": {
        "type": "mapper_parsing_exception",
        "reason": "Could not dynamically add mapping for field [GeoLocation.coordinates]. Existing mapping for [GeoLocation] must be of type object but found [geo_shape]."
      },
      "status": 400
    }

Thank you very much for the reply.

4

Easiest might be to reindex to a temporary index, fix the mapping on your main index, then reindex back there and your data will have the correct mapping.

You can also include your script in the reindex. Something like this.

POST _reindex
{
  "source": {
    "index": "eventanalytics"
  },
  "dest": {
    "index": "temp-index"
  },
  "script": {
    "source": "YOUR SCRIPT"
  }
}
1 Like
5

You could just add another field that is a geo_point... then use a variation of your script in the first post to correctly fill it then use that field on the map.

2 Likes
6

oh thats so simple its brilliant... I will give that a try!

I think i am still stuck on how to accurately define a geo_point field variable within 'PAINLESS'. I find it pretty painful at times :wink:

I think just the array i have is all the geo_point requires [lon,lat]. So I will give that a shot with a small sample in my test environment.

Thanks to you both.

7

There are multiple ways to format it.

"GeoLocation": "41.12,-71.34" might be the easiest when working with painless. It doesn't have to be an array.

8

geo_point and geo_shape are different elasticsearch types. geo_shape accepts data input in GeoJSON and WKT formats as documented here.

If you are dealing only with point geometries, it's definitely easier for you to change your mapping type to geo_point and then select an input format that suits better your script, as @aaron-nimocks and @stephenb suggested.

9

In working on this further taking your assistance to heart... i have tested my script part and just set the values in the groups to a 'string' variable that pre-existed. Here is an example:

COORDINATES
-23.661306;-46.506833

LOCATION
-46.506833,-23.661306

So my parsing with matcher() works that i can tell. I created a new variable of type 'geo_point'. And wish to use the script to insert previously parsed/stored data stored in COORDINATES into this new value.

 "script": "def m = /(.*)\\;(.*)/.matcher(ctx._source.ICTCOORDINATES); if ( m.matches() ) { ctx._source['location_v2'] = '[' + m.group(2) + ',' + m.group(1) + ']'; }"

And i get the following error string:

{
      "index": "eventanalytics-2021.11.11",
      "type": "_doc",
      "id": "6543EH0BiSnVwH86wsGi",
      "cause": {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [location_v2] of type [geo_point]",
        "caused_by": {
          "type": "parse_exception",
          "reason": "latitude must be a number"
        }
      },
      "status": 400
    }

I have tried doing Integer.parseInt(m.group(1)) with the values but it doesn't like that either. With this error:

    "script": "def m = /(.*)\\;(.*)/.matcher(ctx._source.ICTCOORDINATES);  if ( m.matches() ) { ctx._source['location_v2'] = '\"[ ' + Integer.parseInt(m.group(2)) + ',' +  Integer.parseInt(m.group(1)) + ']\"'; }",
    "lang": "painless",
    "caused_by": {
      "type": "number_format_exception",
      "reason": "For input string: \"-46.506833\""
    }

I am just curious if there is something i am doing natively wrong and easily fixed? I know i have the COORDINATES values stored as a string - so the error about not being an integer makes sense - no matter what i have tried it won't take it.

I have tried a simple ",<lat", "[lon,>lat]", etc - all seems to give the same error about int required basically.

Many thanks for all the assistance.

10

Your parsed number is not an integer but a float, so you likely need to use Float.parseFloat() to process the coordinate.

11

I cannot believe i did this...

Thank you for pointing out such a simple mistake! All working like a charm now! Thanks to everyone for the assist!

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.