Hey All,
I have a question about the internal implementation of geo hashes and
distance filters. Here is my current understanding, I'm struggling to
figure out how to apply these to our queries internally in ES.
Using bool queries are very efficient. Internally they
perform bitmap union, intersection, and subtraction for very fast candidate
aggregation per term.
Geo distance filters are then run on the results of the candidates from the
bitmap logic. Each document must be evaluated individually in memory.
Obviously for large documents sets from the bitmap evaluation, this is
inefficient.
What happens when someone only gives our application a geo distance query?
To make this more efficient, I would like to use geo hashing. ES seems to
have geo hashing built in, but it's documented as filter. For instance, I
envision the following workflow internally in ES.
User searches for all matches within 2k of their current location
Use a geohash to create a hash that will encapsulate all points within
2k of their current location
Use the bool query with this geo hash to narrow the candidate result set
Apply the distance filter to these candidates to get more accurate
results.
However, when reading the documentation on searching geo hashing, it's
still a filter. Internally, does it use geohasing and the fast bitmaps
since it's a string match, then filter, or is it all filters and the hash
is evaluated in memory for all documents?
On Friday, November 14, 2014 9:29:54 AM UTC-7, Todd Nine wrote:
Hey All,
I have a question about the internal implementation of geo hashes and
distance filters. Here is my current understanding, I'm struggling to
figure out how to apply these to our queries internally in ES.
Using bool queries are very efficient. Internally they
perform bitmap union, intersection, and subtraction for very fast candidate
aggregation per term.
Geo distance filters are then run on the results of the candidates from
the bitmap logic. Each document must be evaluated individually in memory.
Obviously for large documents sets from the bitmap evaluation, this is
inefficient.
What happens when someone only gives our application a geo distance query?
To make this more efficient, I would like to use geo hashing. ES seems to
have geo hashing built in, but it's documented as filter. For instance, I
envision the following workflow internally in ES.
User searches for all matches within 2k of their current location
Use a geohash to create a hash that will encapsulate all points within
2k of their current location
Use the bool query with this geo hash to narrow the candidate result set
Apply the distance filter to these candidates to get more accurate
results.
However, when reading the documentation on searching geo hashing, it's
still a filter. Internally, does it use geohasing and the fast bitmaps
since it's a string match, then filter, or is it all filters and the hash
is evaluated in memory for all documents?
On Friday, November 14, 2014 9:29:54 AM UTC-7, Todd Nine wrote:
Hey All,
I have a question about the internal implementation of geo hashes and
distance filters. Here is my current understanding, I'm struggling to
figure out how to apply these to our queries internally in ES.
Using bool queries are very efficient. Internally they
perform bitmap union, intersection, and subtraction for very fast candidate
aggregation per term.
Geo distance filters are then run on the results of the candidates from
the bitmap logic. Each document must be evaluated individually in memory.
Obviously for large documents sets from the bitmap evaluation, this is
inefficient.
What happens when someone only gives our application a geo distance query?
To make this more efficient, I would like to use geo hashing. ES seems to
have geo hashing built in, but it's documented as filter. For instance, I
envision the following workflow internally in ES.
User searches for all matches within 2k of their current location
Use a geohash to create a hash that will encapsulate all points within
2k of their current location
Use the bool query with this geo hash to narrow the candidate result set
Apply the distance filter to these candidates to get more accurate
results.
However, when reading the documentation on searching geo hashing, it's
still a filter. Internally, does it use geohasing and the fast bitmaps
since it's a string match, then filter, or is it all filters and the hash
is evaluated in memory for all documents?
+1
Yeah, having geo hashing done internally would be awesome. Kinda confused
since it does only seem to be documented as a filter so any answer would be
helpful.
Thanks!
On Friday, November 14, 2014 8:29:54 AM UTC-8, Todd Nine wrote:
Hey All,
I have a question about the internal implementation of geo hashes and
distance filters. Here is my current understanding, I'm struggling to
figure out how to apply these to our queries internally in ES.
Using bool queries are very efficient. Internally they
perform bitmap union, intersection, and subtraction for very fast candidate
aggregation per term.
Geo distance filters are then run on the results of the candidates from
the bitmap logic. Each document must be evaluated individually in memory.
Obviously for large documents sets from the bitmap evaluation, this is
inefficient.
What happens when someone only gives our application a geo distance query?
To make this more efficient, I would like to use geo hashing. ES seems to
have geo hashing built in, but it's documented as filter. For instance, I
envision the following workflow internally in ES.
User searches for all matches within 2k of their current location
Use a geohash to create a hash that will encapsulate all points within
2k of their current location
Use the bool query with this geo hash to narrow the candidate result set
Apply the distance filter to these candidates to get more accurate
results.
However, when reading the documentation on searching geo hashing, it's
still a filter. Internally, does it use geohasing and the fast bitmaps
since it's a string match, then filter, or is it all filters and the hash
is evaluated in memory for all documents?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.