How to define a new geo_shape field using Painless

[raposa (Rich Raposa)]

I have two float fields that represent a latitude and longitude. I want to perform an _update_by_query and add a new geo_shape field to each document using the values from the lat/lon fields.

It works easily enough for a geo_point:

POST my_index/_update_by_query
{
  "script" : {
    "inline" : """
ctx._source.area.location=[ctx._source.area.longitude, ctx._source.area.latitude]
"""
  }
}

But I cannot figure out the syntax for a geo_shape field. The following code does not work:

ctx._source.area.geo_shape['coordinates']=[ctx._source.area.longitude, ctx._source.area.latitude];
ctx._source.area.geo_shape['type']="point";

A null_pointer_exception is thrown on the geo_shape reference:

"type": "script_exception",
    "reason": "runtime error",
    "script_stack": [
      "ctx._source.area.geo_shape['coordinates']=[ctx._source.area.longitude, ctx._source.area.latitude];\n",
      "                ^---- HERE"
    ],
    "script": "ctx._source.area.geo_shape['coordinates']=[ctx._source.area.longitude, ctx._source.area.latitude];\nctx._source.area.geo_shape['type']=\"point\";",
    "lang": "painless",
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": null
    }

The mapping looks like:

"properties": {
          "area": {
            "properties": {
              "geo_shape": {
                "type": "geo_shape"
              },
              "latitude": {
                "type": "float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "float"
              },
              ...

Anyone tried this before?

[rjernst (Ryan Ernst)]

The ctx._source is a Map. It will not auto create fields. So you would either have to create sub maps (containing area and geo_shape keys), or just create a single field from the root (the dots will automatically be handled when indexing). Eg:

ctx._source['area.geo_shape.coordinates'] = // whatever

You should also aware that coordinates for a geo shape must be in an array, but what you have in your example is a geo json array, so you need to double it. Also not that this will never work, you cannot set mappings inside a document:

ctx._source.area.geo_shape['type']="point";

Finally, why are you using a geo_shape for a single point? You should use a geo_point.

[raposa (Rich Raposa)]

I am trying to use the geo_shape query to determine if a point is located within a preindexed set of geo_shapes. geo_point doesn't work in that scenario.

FWIW, if the document already has a geo_shape field defined, I can use Painless to update the coordinates and the type. The following code works fine:

PUT test/doc/1
{
  "lon" : -87.942655,
  "lat" : 42.034714,
  "my_geo_shape" : {
    "coordinates" : [0,0],
    "type" : "point"
  }
}

POST test/_update_by_query
{
  "script" : {
    "inline": """
ctx._source.my_geo_shape.coordinates = [ctx._source.lon,ctx._source.lat];
ctx._source.my_geo_shape.type="point";
"""    
  }
}

What I really want is for my script to add a new geo_shape field to the doc, but if I run the _update_by_query above on the following document, I get a null_pointer_exception:

PUT test/doc/2
{
  "lon" : -87.942655,
  "lat" : 42.034714
}

What is the syntax for adding a new geo_shape field to a document in Painless?

[raposa (Rich Raposa)]

FWIW: I could not figure out the Painless syntax, so I just moved the work back to Logstash and reindexed. Here is what the Logstash config looked like:

   if(![area][latitude] or ![area][longitude]) {
      mutate {
      	 replace => {"[area][latitude]" => "0"}
   	     replace => {"[area][longitude]" => "0"}
	  }
   }
   
   mutate {
   	add_field => {
   		"[area][my_geoshape][coordinates]" => ["%{[area][longitude]}","%{[area][latitude]}"]
   		"[area][my_geoshape][type]" => "point"
    }
   }

[rjernst (Ryan Ernst)]

Did you try what I suggested earlier, using the full key instead of assuming auto object creation?

ctx._source['area.geo_shape.coordinates'] = [ctx._source.lon, ctx._source.lat]
ctx._source['area.geo_shape.type'] = 'point'

Also, I misread your "type" earlier, thinking it was trying to set the mappings, but in fact it is just a part of the geo shape.

[raposa (Rich Raposa)]

I did try that. It generates the following error:

"cause": {
        "type": "mapper_parsing_exception",
        "reason": "Could not dynamically add mapping for field [area.geo_shape.coordinates]. Existing mapping for [area.geo_shape] must be of type object but found [geo_shape]."
      }

[rjernst (Ryan Ernst)]

Ok, then try:

ctx._source['area.geo_shape'] = [
  'type': 'point',
  'coordinates': [ctx._source.lon, ctx._source.lat]
]

[raposa (Rich Raposa)]

Brilliant! That worked perfectly. Thanks for helping out with this!