Geoquery Against Pre-Indexed Shape is Failing [Second Try]

Waldo_Jaquith · January 4, 2015, 7:53pm

Folks,

[I sent the following on December 7, and got no reply. I’m sending it again and, if nobody tells me otherwise, I’ll assume that this is a bug, and file a report accordingly.]

I’m trying to run a geo_shape filter query using a pre-indexed shape, and it’s failing for reasons that I cannot identify. This is with Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at https://gist.github.com/waldoj/004f77755179d862fe3a.

The pre-indexed shapes are GeoJSON converted from Census Bureau municipal geodata. The indexed documents are state corporate registration records, each of which have an address that I’ve geocoded into a latitude and longitude (a field named “location”). The indexed shapes are in a "geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I specify "path": “geometry”, which generates this error:

"Failed to find geo_shape field [location]].”

Based on the error, it looks like Elasticsearch is looking for shapes that fall within the pre-indexed shape, rather than points, and so while the indexed documents have a geo_point field named “location,” there’s no geo_shape field of that name. FWIW, I’m following the instructions in the manual at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape.

I’d be grateful to anybody who could offer some guidance. Thank you.

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data Institute

202-719-5315

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org.
For more options, visit https://groups.google.com/d/optout.

jprante · January 4, 2015, 11:27pm

I think you already found the answer so I hesitate to comment. If you set
"location" to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.

Best,

Jörg

On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith waldo@usodi.org wrote:

Folks,

[I sent the following on December 7, and got no reply. I’m sending it again and, if nobody tells me otherwise, I’ll assume that this is a bug, and file a report accordingly.]

I’m trying to run a geo_shape filter query using a pre-indexed shape, and
it’s failing for reasons that I cannot identify. This is with Elasticsearch
v1.4.1-1. I’ve documented how to reproduce the problem at <
A demonstration of the Elasticsearch query that yields the error "Failed to find geo_shape field [location]]." This is a search of a collection of documents—each of which contain a pair of coordinates—against a pre-indexed collection of shapes. For simplicity's sake, I've included just one shape and just one document. This is with Elasticsearch v1.4.1-1 on a fresh EC2 instance, running Amazon Linux . · GitHub>.

The pre-indexed shapes are GeoJSON converted from Census Bureau municipal
geodata. The indexed documents are state corporate registration records,
each of which have an address that I’ve geocoded into a latitude and
longitude (a field named “location”). The indexed shapes are in a
"geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I
specify "path": “geometry”, which generates this error:

"Failed to find geo_shape field [location]].”

Based on the error, it looks like Elasticsearch is looking for shapes
that fall within the pre-indexed shape, rather than points, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at <
Elasticsearch Platform — Find real-time answers at scale | Elastic

.

I’d be grateful to anybody who could offer some guidance. Thank you.

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHuSsEkNEwwLnM-T4Kp-zV6Dku%3DZpwf9J9s3%3DR%3DvqF1ww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Peter_Johnson_2 · January 5, 2015, 2:15pm

Jörg is correct in saying that the 'geo_shape' filter only supports finding
shapes within shapes and not points within shapes.

It would be great if there was a filter which did support finding all
points within a pre-indexed shape!

You may want to open an issue regarding extending the 'geo-polygon' filter
to support 'indexed_shape' instead of just 'points'.

As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.

I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?

gist.github.com

https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226

elasticsearch.sh

# Remove old data
curl -XDELETE "http://localhost:9200/shapes"
curl -XDELETE "http://localhost:9200/businesses"

# Create shapes index, with mapping
curl -XPOST "http://localhost:9200/shapes/" -d '
{
  "shapes" : {
    "mappings" : {
      "municipalities" : {

This file has been truncated. show original

Feel free to use my minimal testcase as a template to make your debugging
easier:

gist.github.com

https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e

gis-pre-indexed.sh

#!/bin/bash

################################################
# Pre-indexed shapes using geo_shape filter
################################################

ES='localhost:9200';

# Remove old data
curl -XDELETE "$ES/shapes?pretty=true";

This file has been truncated. show original

-P

On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:

I think you already found the answer so I hesitate to comment. If you set
"location" to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.

Best,

Jörg

On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith <wa...@usodi.org
<javascript:>> wrote:

Folks,

[I sent the following on December 7, and got no reply. I’m sending it again and, if nobody tells me otherwise, I’ll assume that this is a bug, and file a report accordingly.]

I’m trying to run a geo_shape filter query using a pre-indexed shape, and
it’s failing for reasons that I cannot identify. This is with Elasticsearch
v1.4.1-1. I’ve documented how to reproduce the problem at <
A demonstration of the Elasticsearch query that yields the error "Failed to find geo_shape field [location]]." This is a search of a collection of documents—each of which contain a pair of coordinates—against a pre-indexed collection of shapes. For simplicity's sake, I've included just one shape and just one document. This is with Elasticsearch v1.4.1-1 on a fresh EC2 instance, running Amazon Linux . · GitHub>.

The pre-indexed shapes are GeoJSON converted from Census Bureau municipal
geodata. The indexed documents are state corporate registration records,
each of which have an address that I’ve geocoded into a latitude and
longitude (a field named “location”). The indexed shapes are in a
"geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I
specify "path": “geometry”, which generates this error:

"Failed to find geo_shape field [location]].”

Based on the error, it looks like Elasticsearch is looking for shapes
that fall within the pre-indexed shape, rather than points, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at <
Elasticsearch Platform — Find real-time answers at scale | Elastic

.

I’d be grateful to anybody who could offer some guidance. Thank you.

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3ea4e571-e192-4136-b860-65c19801193c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Waldo_Jaquith · January 6, 2015, 2:52am

This is fascinating—it hadn't crossed my mind that I might be doing
something that simply wasn't supported by Elasticsearch. As many times as I
read the documentation, I think I was reading for what I wanted it to do,
rather than as a blank slate. Perhaps I'm trying to do something that is
supported, but I'm doing it wrong?

This is for all businesses registered in a U.S. state (Virginia), to
identify which ones are registered in a given county or city (a sub-unit of
a state). The UK equivalent of this would be a list of all businesses
registered in the East Midlands, and wanting to identify all of the
businesses in Nottingham. I want somebody running a search for a business
name to be able to limit that search to a given county or city. Perhaps I'm
doing that all wrong, in terms of how Elasticsearch works. (At present, I'm
doing this by including the shapes in my HTML, rather than pre-indexing
them.) Is there a more Elasticsearch-y way to accomplish this, or is it
really best for me to open an issue proposing such a new feature?

My apologies for the errors in my gist! Thank you for forking it and fixing
them. This is my first time putting together a complete test case of a
problem for a mailing list, so when the test case failed, it only served to
(wrongly) reinforce that I'd correctly reproduced the problem!

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315

On Monday, January 5, 2015 9:15:51 AM UTC-5, Peter Johnson wrote:

Jörg is correct in saying that the 'geo_shape' filter only supports
finding shapes within shapes and not points within shapes.

It would be great if there was a filter which did support finding all
points within a pre-indexed shape!

You may want to open an issue regarding extending the 'geo-polygon' filter
to support 'indexed_shape' instead of just 'points'.

Elasticsearch Platform — Find real-time answers at scale | Elastic

As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.

I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?

A demonstration of the Elasticsearch query that yields the error "Failed to find geo_shape field [location]]." This is a search of a collection of documents—each of which contain a pair of coordinates—against a pre-indexed collection of shapes. For simplicity's sake, I've included just one shape and just one document. This is with Elasticsearch v1.4.1-1 on a fresh EC2 instance, running Amazon Linux . · GitHub

Feel free to use my minimal testcase as a template to make your debugging
easier:
Pre-indexed shapes using geo_shape filter (not working) · GitHub

-P

On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:

I think you already found the answer so I hesitate to comment. If you set
"location" to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.

Best,

Jörg

On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org wrote:

Folks,

[I sent the following on December 7, and got no reply. I’m sending it again and, if nobody tells me otherwise, I’ll assume that this is a bug, and file a report accordingly.]

I’m trying to run a geo_shape filter query using a pre-indexed shape,
and it’s failing for reasons that I cannot identify. This is with
Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at <
A demonstration of the Elasticsearch query that yields the error "Failed to find geo_shape field [location]]." This is a search of a collection of documents—each of which contain a pair of coordinates—against a pre-indexed collection of shapes. For simplicity's sake, I've included just one shape and just one document. This is with Elasticsearch v1.4.1-1 on a fresh EC2 instance, running Amazon Linux . · GitHub>.

The pre-indexed shapes are GeoJSON converted from Census Bureau
municipal geodata. The indexed documents are state corporate registration
records, each of which have an address that I’ve geocoded into a latitude
and longitude (a field named “location”). The indexed shapes are in a
"geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I
specify "path": “geometry”, which generates this error:

"Failed to find geo_shape field [location]].”

Based on the error, it looks like Elasticsearch is looking for shapes
that fall within the pre-indexed shape, rather than points, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at <
Elasticsearch Platform — Find real-time answers at scale | Elastic

.

I’d be grateful to anybody who could offer some guidance. Thank you.

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7af28b30-933f-4b5c-be65-24800a0f16fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Peter_Johnson_2 · January 6, 2015, 3:15pm

Hey Waldo,

I had a look over some of the code we have for Pelias
https://github.com/pelias/pelias and I managed to get it to work, the
trick being that you need to index the business location as a 'geo-shape'
'point' type:

gist.github.com

https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L50

gis-pre-indexed.sh

#!/bin/bash

################################################
# Pre-indexed shapes using geo_shape filter
################################################

ES='localhost:9200';

# Remove old data
curl -XDELETE "$ES/shapes?pretty=true";

This file has been truncated. show original

gist.github.com

https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L62

gis-pre-indexed.sh

#!/bin/bash

################################################
# Pre-indexed shapes using geo_shape filter
################################################

ES='localhost:9200';

# Remove old data
curl -XDELETE "$ES/shapes?pretty=true";

This file has been truncated. show original

Working example: Pre-indexed shapes using geo_shape filter (working) · GitHub
Visual
representation: map.geojson · GitHub

When we index openstreetmap data in elasticsearch, we first find the
intersection of the lat/lon within multiple indeces containing polygon
information sourced from the quattroshapes project.
The results are then stored as 'admin' properties against each record
http://pelias.mapzen.com/reverse?lat=38.0509&lon=-78.4522&size=20 before
indexing the POI, which means doing a lookup for all places inside polygon
becomes a simple string match rather than a complex query-time computation.

Would love to have a chat some time about collaborating on US polygon data
and sources of open address data.

All the best,
-P

On Tuesday, 6 January 2015 02:52:05 UTC, Waldo Jaquith wrote:

This is fascinating—it hadn't crossed my mind that I might be doing
something that simply wasn't supported by Elasticsearch. As many times as I
read the documentation, I think I was reading for what I wanted it to do,
rather than as a blank slate. Perhaps I'm trying to do something that is
supported, but I'm doing it wrong?

This is for all businesses registered in a U.S. state (Virginia), to
identify which ones are registered in a given county or city (a sub-unit of
a state). The UK equivalent of this would be a list of all businesses
registered in the East Midlands, and wanting to identify all of the
businesses in Nottingham. I want somebody running a search for a business
name to be able to limit that search to a given county or city. Perhaps I'm
doing that all wrong, in terms of how Elasticsearch works. (At present, I'm
doing this by including the shapes in my HTML, rather than pre-indexing
them.) Is there a more Elasticsearch-y way to accomplish this, or is it
really best for me to open an issue proposing such a new feature?

My apologies for the errors in my gist! Thank you for forking it and
fixing them. This is my first time putting together a complete test case of
a problem for a mailing list, so when the test case failed, it only served
to (wrongly) reinforce that I'd correctly reproduced the problem!

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315

On Monday, January 5, 2015 9:15:51 AM UTC-5, Peter Johnson wrote:

Jörg is correct in saying that the 'geo_shape' filter only supports
finding shapes within shapes and not points within shapes.

It would be great if there was a filter which did support finding all
points within a pre-indexed shape!

You may want to open an issue regarding extending the 'geo-polygon'
filter to support 'indexed_shape' instead of just 'points'.

Elasticsearch Platform — Find real-time answers at scale | Elastic

As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.

I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?

A demonstration of the Elasticsearch query that yields the error "Failed to find geo_shape field [location]]." This is a search of a collection of documents—each of which contain a pair of coordinates—against a pre-indexed collection of shapes. For simplicity's sake, I've included just one shape and just one document. This is with Elasticsearch v1.4.1-1 on a fresh EC2 instance, running Amazon Linux . · GitHub

Feel free to use my minimal testcase as a template to make your debugging
easier:
Pre-indexed shapes using geo_shape filter (not working) · GitHub

-P

On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:

I think you already found the answer so I hesitate to comment. If you
set "location" to type geo_point and not to geo_shape, you can not execute
a geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.

Best,

Jörg

On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org wrote:

Folks,

[I sent the following on December 7, and got no reply. I’m sending it again and, if nobody tells me otherwise, I’ll assume that this is a bug, and file a report accordingly.]

I’m trying to run a geo_shape filter query using a pre-indexed shape,
and it’s failing for reasons that I cannot identify. This is with
Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at <
A demonstration of the Elasticsearch query that yields the error "Failed to find geo_shape field [location]]." This is a search of a collection of documents—each of which contain a pair of coordinates—against a pre-indexed collection of shapes. For simplicity's sake, I've included just one shape and just one document. This is with Elasticsearch v1.4.1-1 on a fresh EC2 instance, running Amazon Linux . · GitHub>.

The pre-indexed shapes are GeoJSON converted from Census Bureau
municipal geodata. The indexed documents are state corporate registration
records, each of which have an address that I’ve geocoded into a latitude
and longitude (a field named “location”). The indexed shapes are in a
"geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I
specify "path": “geometry”, which generates this error:

"Failed to find geo_shape field [location]].”

Based on the error, it looks like Elasticsearch is looking for shapes
that fall within the pre-indexed shape, rather than points, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at <
Elasticsearch Platform — Find real-time answers at scale | Elastic

.

I’d be grateful to anybody who could offer some guidance. Thank you.

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/885b73a7-3486-45fa-9aba-fa98319406fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Waldo_Jaquith · January 8, 2015, 3:37am

This is just wonderful, Peter—thank you. (Providing a complete, tested, working example is probably more than I’ve ever done for anybody seeking debugging help!) It looks like I went and made the indexed location data more complicated than it needed to be, ginning up an object / geo_point structure, when all I really needed was "type: point” and “coordinates: -78, 38”. This has been a blocker on a project for so long, I scarcely remember how to proceed from here. My thanks to you and the Elasticsearch community for your help!

And, yes, I’d be happy to talk about US polygon data and sources of open address data. My organization is keenly interested in improving the state of open geodata in the US, and I’m eager to talk to anybody about how U.S. Open Data could help to improve things.

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315

On Jan 6, 2015, at 10:15 AM, Peter Johnson peter@missinglink.co.nz wrote:

Hey Waldo,

I had a look over some of the code we have for Pelias and I managed to get it to work, the trick being that you need to index the business location as a 'geo-shape' 'point' type:
Pre-indexed shapes using geo_shape filter (working) · GitHub
Pre-indexed shapes using geo_shape filter (working) · GitHub

Working example: Pre-indexed shapes using geo_shape filter (working) · GitHub
Visual representation: map.geojson · GitHub

When we index openstreetmap data in elasticsearch, we first find the intersection of the lat/lon within multiple indeces containing polygon information sourced from the quattroshapes project.
The results are then stored as 'admin' properties against each record before indexing the POI, which means doing a lookup for all places inside polygon becomes a simple string match rather than a complex query-time computation.

Would love to have a chat some time about collaborating on US polygon data and sources of open address data.

All the best,
-P

On Tuesday, 6 January 2015 02:52:05 UTC, Waldo Jaquith wrote:
This is fascinating—it hadn't crossed my mind that I might be doing something that simply wasn't supported by Elasticsearch. As many times as I read the documentation, I think I was reading for what I wanted it to do, rather than as a blank slate. Perhaps I'm trying to do something that is supported, but I'm doing it wrong?

This is for all businesses registered in a U.S. state (Virginia), to identify which ones are registered in a given county or city (a sub-unit of a state). The UK equivalent of this would be a list of all businesses registered in the East Midlands, and wanting to identify all of the businesses in Nottingham. I want somebody running a search for a business name to be able to limit that search to a given county or city. Perhaps I'm doing that all wrong, in terms of how Elasticsearch works. (At present, I'm doing this by including the shapes in my HTML, rather than pre-indexing them.) Is there a more Elasticsearch-y way to accomplish this, or is it really best for me to open an issue proposing such a new feature?

My apologies for the errors in my gist! Thank you for forking it and fixing them. This is my first time putting together a complete test case of a problem for a mailing list, so when the test case failed, it only served to (wrongly) reinforce that I'd correctly reproduced the problem!

Best,
Waldo

Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E4F6AC07-3A57-404C-B73D-2DBA3266207C%40usodi.org.
For more options, visit https://groups.google.com/d/optout.