Incorrect geo_shape results in both es6.x and 7.x

Elastic Stack Elasticsearch
1

I'm still seeing the same incorrect shape query behavior that I saw back to ES 1.7 after starting fresh on es 6.7 and 7.x (hosted by elastic.co). Here is the relevant data:

Documents with WKT shape data for NY and CT states that look like this:
NY:

13%20PM
Screen Shot 2019-08-15 at 12.28.13 PM.jpg1384×1104 553 KB

CT:
19%20PM
Screen Shot 2019-08-15 at 12.28.19 PM.png1378×1102 483 KB

and I'm using this query to look for shapes that contain this point, clearly within CT and not within NY state:

GET /senate/_search
{
"query":{
    "bool": {
        "must": {
            "match_all": {}
        },
        "filter": {
            "geo_shape": {
                "shape": {
                    "shape": {
                        "type": "point",
                        "coordinates" : [-73.572866,41.142307]
                    }
                }
            }
        }
    }
}}

The point is located in North Stamford on a map, and to be honest not really that close to NY.

43%20PM
Screen Shot 2019-08-15 at 12.29.43 PM.jpg2050×1458 915 KB

This query returns documents with BOTH NY and CT shapes whereas it should only return documents with CT shapes.

@spinscale any idea why this is happening or how I can get actual geo_shape queries that aren't some approximation of a shape? I don't care if it takes an extra 500ms, I just want the queries to return the accurate data that they clearly should.

2

Perhaps you need to use the relation INTERSECTS

"relation": "intersects"

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html

Note / edit yes that is supposed to be the default but you might try being explicit

3

Hi Stephen,

Same incorrect results setting the relation specifically to "intersect".

4

Asking internally....

What happens with..

"relation": "contains"

Apologies for not being as helpful as I would like....

5

Same results for "contains" (and "within" just for fun). Thanks for asking around.

6

Can you post your mapping for the state shape data

7

Relevant shape property is this one:

	"properties":{
		"shape":{
			"type":"geo_shape",
			"strategy" : "recursive"
		}
	}
8

So here is my working version.

What I did is create 2 simple polygons called the CT and NY (see images at the bottom)
Then created 3 queries 1 in CT, 1 in NY and 1 in neither all are working as expected.
I think I would stay away from the strategy stuff as it is all deprecated.

See if you can repeat

Here is all my code........note I am only returning the name with the "_source": ["name"], directive in the query so as not to return big polygons... you can take that out if you want to see my polygon results

# GEO STUFF
DELETE /regions

PUT /regions
{
  "mappings": {
    "properties": {
      "name" : {"type" : "text"},
      "location": {
        "type": "geo_shape"
      }
    }
  }
}

GET /regions/_mapping

{
  "regions" : {
    "mappings" : {
      "properties" : {
        "location" : {
          "type" : "geo_shape"
        },
        "name" : {
          "type" : "text"
        }
      }
    }
  }
}

POST /regions/_doc
{
  "name": "CT",
  "location": {
    "type": "polygon",
    "coordinates": [
      [
        [
          -73.7268427,
          41.1024415
        ],
        [
          -73.6533716,
          41.0076869
        ],
        [
          -73.3979395,
          41.0931274
        ],
        [
          -73.4830835,
          41.2115246
        ],
        [
          -73.7268427,
          41.1024415
        ]
      ]
    ]
  }
}

POST /regions/_doc
{
  "name": "NY",
  "location": {
    "type": "polygon",
    "coordinates": [
      [
        [
          -73.7268427,
          41.1024415
        ],
        [
          -73.4849353,
          41.2137893
        ],
        [
          -73.5515399,
          41.2974142
        ],
        [
          -73.797359,
          41.2143058
        ],
        [
          -73.7268427,
          41.1024415
        ]
      ]
    ]
  }
}

# Point in CT
GET /regions/_search
{
  "_source": [ "name" ],
  "query": {
    "geo_shape": {
      "location": {
        "shape": {
          "type": "point",
          "coordinates": [
            -73.5576296,
            41.1012336
          ]
        }
      }
    }
  }
}


# Point in NY
GET /regions/_search
{
  "_source": ["name"],
  "query": {
    "geo_shape": {
      "location": {
        "shape": {
          "type": "point",
          "coordinates": [
            -73.6490436,
            41.2075906
          ]
        }
      }
    }
  }
}


# Point in Niether
GET /regions/_search
{
  "_source": ["name"],
  "query": {
    "geo_shape": {
      "location": {
        "shape": {
          "type": "point",
          "coordinates": [
            -73.7190814,
            41.3293904
          ]
        }
      }
    }
  }
}


# Point in CT with bool
GET /regions/_search
{
  "_source": [
    "name"
  ],
  "query": {
    "bool": {
      "must": {
        "match_all": {}
      },
      "filter": {
        "geo_shape": {
          "location": {
            "shape": {
              "type": "point",
              "coordinates": [
                -73.5576296,
                41.1012336
              ]
            }
          }
        }
      }
    }
  }
}

# NY with Bool 
GET /regions/_search
{
  "_source": ["name"],
  "query": {
    "bool": {
      "must": {
        "match_all": {}
      },
      "filter": {
        "geo_shape": {
          "location": {
            "shape": {
              "type": "point",
              "coordinates": [
                -73.6490436,
                41.2075906
              ]
            }
          }
        }
      }
    }
  }
}



# Neither
GET /regions/_search
{
  "_source": [
    "name"
  ],
  "query": {
    "bool": {
      "must": {
        "match_all": {}
      },
      "filter": {
        "geo_shape": {
          "location": {
            "shape": {
              "type": "point",
              "coordinates": [
                -73.7190814,
                41.3293904
              ]
            }
          }
        }
      }
    }
  }
}

Here are the polygons / images
CT

41%20PM
Screen Shot 2019-08-15 at 9.32.41 PM.jpg3232×1808 1.19 MB

NY

11%20PM
Screen Shot 2019-08-15 at 9.36.11 PM.jpg3024×1836 1.15 MB

Edit : Fixed Code... all working now

Note : I did a little more complex NY with it "surrounding" CT and it works ... any possibility that there is an issue importing the WKTs? You know in 7.3 you can easily display them in the Maps Application example (The Neither Point is now in NY with this example)

22%20PM
Screen Shot 2019-08-15 at 11.19.22 PM.jpg3088×1792 1.18 MB

9

Hi Stephen,

There are plenty of points that appear to work correctly in my version as well, something like -73.554212, 41.053923 gives me results from inside CT only. My question is why there are some points that do not work correctly with these non-overlapping shapes.

WKTs for NY and CT follow:

POLYGON((-73.343806 45.013027,-73.332852 44.804903,-73.387622 44.618687,-73.294514 44.437948,-73.321898 44.246255,-73.436914 44.043608,-73.349283 43.769761,-73.404052 43.687607,-73.245221 43.523299,-73.278083 42.833204,-73.267129 42.745573,-73.508114 42.08834,-73.486206 42.050002,-73.55193 41.294184,-73.48073 41.21203,-73.727192 41.102491,-73.655992 40.987475,-73.22879 40.905321,-73.141159 40.965568,-72.774204 40.965568,-72.587988 40.998429,-72.28128 41.157261,-72.259372 41.042245,-72.100541 40.992952,-72.467496 40.845075,-73.239744 40.625997,-73.562884 40.582182,-73.776484 40.593136,-73.935316 40.543843,-74.022947 40.708151,-73.902454 40.998429,-74.236547 41.14083,-74.69661 41.359907,-74.740426 41.431108,-74.89378 41.436584,-75.074519 41.60637,-75.052611 41.754247,-75.173104 41.869263,-75.249781 41.863786,-75.35932 42.000709,-79.76278 42.000709,-79.76278 42.252649,-79.76278 42.269079,-79.149363 42.55388,-79.050778 42.690804,-78.853608 42.783912,-78.930285 42.953697,-79.012439 42.986559,-79.072686 43.260406,-78.486653 43.375421,-77.966344 43.369944,-77.75822 43.34256,-77.533665 43.233021,-77.391265 43.276836,-76.958587 43.271359,-76.695693 43.34256,-76.41637 43.523299,-76.235631 43.528776,-76.230154 43.802623,-76.137046 43.961454,-76.3616 44.070993,-76.312308 44.196962,-75.912491 44.366748,-75.764614 44.514625,-75.282643 44.848718,-74.828057 45.018503,-74.148916 44.991119,-73.343806 45.013027))

POLYGON((-73.053528 42.039048,-71.799309 42.022617,-71.799309 42.006186,-71.799309 41.414677,-71.859555 41.321569,-71.947186 41.338,-72.385341 41.261322,-72.905651 41.28323,-73.130205 41.146307,-73.371191 41.102491,-73.655992 40.987475,-73.727192 41.102491,-73.48073 41.21203,-73.55193 41.294184,-73.486206 42.050002,-73.053528 42.039048))

10

Can you provide several points that do not work actual and expected results. If / when I can... I will take a look.

11

Points that work correctly:
-73.554212, 41.053923 (stamford, CT)
-73.624998,41.047451 (greenwich, CT)
-73.708180,41.126757 (armonk, NY)

Points that do not work correctly:
-73.572866,41.142307 (north stamford, CT)
-73.681591,41.098201 (greenwich, CT)
-73.683025,41.017221 (rye brook, NY)

Some points that work are close to the border, some points that do not work are farther away. It definitely seems like the query happens on a different (possibly simplified?) shape than the one you would expect from viewing the WKT shape.

12

@nickoneill
Interesting... I am having the exact opposite experience using 7.3 .... I put in the WKT Regions and all the searches you provided return correct results and I can not get a bad result randomly and when I zoom down within meters it seems the results are still good until I place a point almost directly on top of the line.. that is not to say that there is not an issue.

DELETE /regions

PUT /regions
{
  "mappings": {
    "properties": {
      "name" : {"type" : "text"},
      "location": {
        "type": "geo_shape"
      }
    }
  }
}

POST /regions/_doc
{
  "name": "NY",
  "location": "POLYGON((-73.343806 45.013027,-73.332852 44.804903,-73.387622 44.618687,-73.294514 44.437948,-73.321898 44.246255,-73.436914 44.043608,-73.349283 43.769761,-73.404052 43.687607,-73.245221 43.523299,-73.278083 42.833204,-73.267129 42.745573,-73.508114 42.08834,-73.486206 42.050002,-73.55193 41.294184,-73.48073 41.21203,-73.727192 41.102491,-73.655992 40.987475,-73.22879 40.905321,-73.141159 40.965568,-72.774204 40.965568,-72.587988 40.998429,-72.28128 41.157261,-72.259372 41.042245,-72.100541 40.992952,-72.467496 40.845075,-73.239744 40.625997,-73.562884 40.582182,-73.776484 40.593136,-73.935316 40.543843,-74.022947 40.708151,-73.902454 40.998429,-74.236547 41.14083,-74.69661 41.359907,-74.740426 41.431108,-74.89378 41.436584,-75.074519 41.60637,-75.052611 41.754247,-75.173104 41.869263,-75.249781 41.863786,-75.35932 42.000709,-79.76278 42.000709,-79.76278 42.252649,-79.76278 42.269079,-79.149363 42.55388,-79.050778 42.690804,-78.853608 42.783912,-78.930285 42.953697,-79.012439 42.986559,-79.072686 43.260406,-78.486653 43.375421,-77.966344 43.369944,-77.75822 43.34256,-77.533665 43.233021,-77.391265 43.276836,-76.958587 43.271359,-76.695693 43.34256,-76.41637 43.523299,-76.235631 43.528776,-76.230154 43.802623,-76.137046 43.961454,-76.3616 44.070993,-76.312308 44.196962,-75.912491 44.366748,-75.764614 44.514625,-75.282643 44.848718,-74.828057 45.018503,-74.148916 44.991119,-73.343806 45.013027))"
}

POST /regions/_doc
{
  "name": "CT",
  "location": "POLYGON((-73.053528 42.039048,-71.799309 42.022617,-71.799309 42.006186,-71.799309 41.414677,-71.859555 41.321569,-71.947186 41.338,-72.385341 41.261322,-72.905651 41.28323,-73.130205 41.146307,-73.371191 41.102491,-73.655992 40.987475,-73.727192 41.102491,-73.48073 41.21203,-73.55193 41.294184,-73.486206 42.050002,-73.053528 42.039048))"
  
}

All these searches return correct results a created them as points to view as well

-73.554212, 41.053923 (stamford, CT)
-73.624998,41.047451 (greenwich, CT)
-73.708180,41.126757 (armonk, NY)

These Points work correctly as well
-73.572866,41.142307 (north stamford, CT)
-73.681591,41.098201 (greenwich, CT)
-73.683025,41.017221 (rye brook, NY)

Using this query and substituting in each in turn I got correct results for each. I also put in a number very close to the borders and they were correct.

north stamford : correct

GET /regions/_search
{
  "_source": [
    "name"
  ],
  "query": {
    "bool": {
      "must": {
        "match_all": {}
      },
      "filter": {
        "geo_shape": {
          "location": {
            "shape": {
              "type": "point",
              "coordinates": [
                -73.572866,
                41.142307
              ]
            }
          }
        }
      }
    }
  }
}

results for north stamford etc

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "regions",
        "_type" : "_doc",
        "_id" : "GFC8nWwBvL0x7al8Ij6K",
        "_score" : 1.0,
        "_source" : {
          "name" : "CT"
        }
      }
    ]
  }
}

Not sure what to tell you / think at this point. This is all on 7.3...

23%20PM
Screen Shot 2019-08-16 at 9.46.23 PM.jpg1636×1286 285 KB

13

Thanks for testing again Stephen, this works as expected without the strategy for the geo_shape mapping set to recursive.

I was hoping to use the circle shape for some queries. Without recursive, using circle in queries returns "CIRCLE geometry is not supported." I can work around this for now.

Do you know the technical details of why recursive makes this behave differently?

14

No that is beyond my spatial knowledge at this point (minor pun) but if I have time I will look into it. So is the circle / radius search within the larger polygon the important aspect of what you are trying to accomplish. often I see circle / radius searches that are larger than smaller polygons like property outlines looking for intersection , seems like you are trying to do a small circle/ radius within the larger polygons is that correct? I think I'm lacking insight on what that accomplishes for your use case. What is your expectation when the circle intersects more than 1 polygon.

Note: it does look like we are in the middle of a major re-engineering of the spatial engine that is kind of indicated in the docs, some features are still there and marked deprecated as the new engine is moving towards more full features / feature parity with improved performance and accuracy.

15

It's not really important to use circle for me, just one of the use cases I was testing out.

I'm all set here, thank you for your perseverance helping me with this issue @stephenb!

1 Like