Query never returns

mattweber · February 18, 2012, 6:43pm

I have a need to make sure all hits from a specific field in a specific type are always the highest ranked documents. To do this, I have used a constant score query with a really high boost value against this type. At the same time, I still want to search across other types and fields in my index. I have crafted a boolean query with 2 should clauses. The first contains the constant score query and the second contains a query string query. Everything works as expected until my constant score query uses a filter. When I specify a filter, the query locks up and never returns a response.

The only reason I decided to use a filter was for better performance. If I remove the 2nd clause from the boolean query, the search executes just fine. If I leave the 2nd clause, and remove the filter from the constant score query, it also returns fine. If I do the constant score filter query by itself, it also returns the expected query.

Here is a gist with the simplest example of the problem:

gist.github.com

https://gist.github.com/mattweber/1860568

failed query (has filter)

curl -XGET 'http://localhost:9200/test/_search?pretty=true' -d '{
    "query":{
        "bool":{
            "should": [
                {
                    "constant_score":{
                        "filter" : {
                            "query":{
                                "query_string":{
                                    "query":"findme",

This file has been truncated. show original

individual querys with filters work

curl -XGET 'http://localhost:9200/test/_search?pretty=true' -d '{
    "query":{
        "bool":{
            "should": [
                {
                    "constant_score":{
                        "filter" : {
                            "query":{
                                "query_string":{
                                    "query":"findme",

This file has been truncated. show original

test docs

curl -XPUT 'http://localhost:9200/test/regular/1' -d '{
    "title" : "Regular Doc1",
    "body" : "findme"
}'

curl -XPUT 'http://localhost:9200/test/regular/2' -d '{
    "title" : "Regular Doc2",
    "body" : "findme"
}'

This file has been truncated. show original

There are more than three files. show original

I am testing with 0.19.0.RC2.

--
Matt Weber
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

kimchy · February 20, 2012, 1:05pm

Regarding the recreation, I ran it against a simple index, and it doesn't lock up, can you provide a recreation that shows the lock up starting fresh (possibly indexing sample docs)?

Regarding boosting specific type, you can use the custom filters score query (http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query.html), and have a term filter on _type that boosts the score if the document matches that type.

On Saturday, February 18, 2012 at 8:43 PM, Matt Weber wrote:

I have a need to make sure all hits from a specific field in a specific type are always the highest ranked documents. To do this, I have used a constant score query with a really high boost value against this type. At the same time, I still want to search across other types and fields in my index. I have crafted a boolean query with 2 should clauses. The first contains the constant score query and the second contains a query string query. Everything works as expected until my constant score query uses a filter. When I specify a filter, the query locks up and never returns a response.

The only reason I decided to use a filter was for better performance. If I remove the 2nd clause from the boolean query, the search executes just fine. If I leave the 2nd clause, and remove the filter from the constant score query, it also returns fine. If I do the constant score filter query by itself, it also returns the expected query.

Here is a gist with the simplest example of the problem:
failed query (has filter) · GitHub

I am testing with 0.19.0.RC2.

--
Matt Weber
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

mattweber · February 20, 2012, 3:19pm

The gist has the sample docs and sample query that locks it up. Just to double check I installed a fresh copy 0.19.0.RC2, started it, copy and pasted the test docs from the gist, then copy and pasted the query from the gist and it locked up.

gist.github.com

https://gist.github.com/mattweber/1860568

failed query (has filter)

curl -XGET 'http://localhost:9200/test/_search?pretty=true' -d '{
    "query":{
        "bool":{
            "should": [
                {
                    "constant_score":{
                        "filter" : {
                            "query":{
                                "query_string":{
                                    "query":"findme",

This file has been truncated. show original

individual querys with filters work

curl -XGET 'http://localhost:9200/test/_search?pretty=true' -d '{
    "query":{
        "bool":{
            "should": [
                {
                    "constant_score":{
                        "filter" : {
                            "query":{
                                "query_string":{
                                    "query":"findme",

This file has been truncated. show original

test docs

curl -XPUT 'http://localhost:9200/test/regular/1' -d '{
    "title" : "Regular Doc1",
    "body" : "findme"
}'

curl -XPUT 'http://localhost:9200/test/regular/2' -d '{
    "title" : "Regular Doc2",
    "body" : "findme"
}'

This file has been truncated. show original

There are more than three files. show original

--
Matt Weber
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Monday, February 20, 2012 at 5:05 AM, Shay Banon wrote:

Regarding the recreation, I ran it against a simple index, and it doesn't lock up, can you provide a recreation that shows the lock up starting fresh (possibly indexing sample docs)?

Regarding boosting specific type, you can use the custom filters score query (http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query.html), and have a term filter on _type that boosts the score if the document matches that type.

On Saturday, February 18, 2012 at 8:43 PM, Matt Weber wrote:

I have a need to make sure all hits from a specific field in a specific type are always the highest ranked documents. To do this, I have used a constant score query with a really high boost value against this type. At the same time, I still want to search across other types and fields in my index. I have crafted a boolean query with 2 should clauses. The first contains the constant score query and the second contains a query string query. Everything works as expected until my constant score query uses a filter. When I specify a filter, the query locks up and never returns a response.

The only reason I decided to use a filter was for better performance. If I remove the 2nd clause from the boolean query, the search executes just fine. If I leave the 2nd clause, and remove the filter from the constant score query, it also returns fine. If I do the constant score filter query by itself, it also returns the expected query.

Here is a gist with the simplest example of the problem:
failed query (has filter) · GitHub

I am testing with 0.19.0.RC2.

--
Matt Weber
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

egaumer · February 20, 2012, 9:36pm

I was able to reproduce this based on the gist. Debug logs don't show
anything but the CPU spikes (when you run the query) and never comes back
down (even when you kill the query). You actually have to restart
elasticsearch.

kimchy · February 21, 2012, 4:29pm

It seems like its stuck in a look within Lucene boolean query, still need to chase it more and figure out what goes wrong, can you open an issue just so we won't loose it?

On Monday, February 20, 2012 at 5:19 PM, Matt Weber wrote:

The gist has the sample docs and sample query that locks it up. Just to double check I installed a fresh copy 0.19.0.RC2, started it, copy and pasted the test docs from the gist, then copy and pasted the query from the gist and it locked up.

failed query (has filter) · GitHub

--
Matt Weber
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Monday, February 20, 2012 at 5:05 AM, Shay Banon wrote:

Regarding the recreation, I ran it against a simple index, and it doesn't lock up, can you provide a recreation that shows the lock up starting fresh (possibly indexing sample docs)?

Regarding boosting specific type, you can use the custom filters score query (http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query.html), and have a term filter on _type that boosts the score if the document matches that type.

On Saturday, February 18, 2012 at 8:43 PM, Matt Weber wrote:

I have a need to make sure all hits from a specific field in a specific type are always the highest ranked documents. To do this, I have used a constant score query with a really high boost value against this type. At the same time, I still want to search across other types and fields in my index. I have crafted a boolean query with 2 should clauses. The first contains the constant score query and the second contains a query string query. Everything works as expected until my constant score query uses a filter. When I specify a filter, the query locks up and never returns a response.

The only reason I decided to use a filter was for better performance. If I remove the 2nd clause from the boolean query, the search executes just fine. If I leave the 2nd clause, and remove the filter from the constant score query, it also returns fine. If I do the constant score filter query by itself, it also returns the expected query.

Here is a gist with the simplest example of the problem:
failed query (has filter) · GitHub

I am testing with 0.19.0.RC2.

--
Matt Weber
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

Ivan · February 21, 2012, 5:17pm

I have seen queries not returning if there are older versions of
Lucene in the classpath. Not every query uses every Lucene
class/method. Certain queries might use a different method, and if the
previous version of Lucene does not have that method, a query to ES
will simply hang with no exception thrown on the client or server
side.

--
Ivan

On Tue, Feb 21, 2012 at 8:29 AM, Shay Banon kimchy@gmail.com wrote:

It seems like its stuck in a look within Lucene boolean query, still need to
chase it more and figure out what goes wrong, can you open an issue just so
we won't loose it?

On Monday, February 20, 2012 at 5:19 PM, Matt Weber wrote:

The gist has the sample docs and sample query that locks it up. Just to
double check I installed a fresh copy 0.19.0.RC2, started it, copy and
pasted the test docs from the gist, then copy and pasted the query from the
gist and it locked up.

failed query (has filter) · GitHub

--
Matt Weber
Sent with Sparrow

On Monday, February 20, 2012 at 5:05 AM, Shay Banon wrote:

Regarding the recreation, I ran it against a simple index, and it doesn't
lock up, can you provide a recreation that shows the lock up starting fresh
(possibly indexing sample docs)?

Regarding boosting specific type, you can use the custom filters score query
(http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query.html),
and have a term filter on _type that boosts the score if the document
matches that type.

On Saturday, February 18, 2012 at 8:43 PM, Matt Weber wrote:

I have a need to make sure all hits from a specific field in a specific type
are always the highest ranked documents. To do this, I have used a constant
score query with a really high boost value against this type. At the same
time, I still want to search across other types and fields in my index. I
have crafted a boolean query with 2 should clauses. The first contains the
constant score query and the second contains a query string query.
Everything works as expected until my constant score query uses a filter.
When I specify a filter, the query locks up and never returns a response.

The only reason I decided to use a filter was for better performance. If I
remove the 2nd clause from the boolean query, the search executes just fine.
If I leave the 2nd clause, and remove the filter from the constant score
query, it also returns fine. If I do the constant score filter query by
itself, it also returns the expected query.

Here is a gist with the simplest example of the problem:
failed query (has filter) · GitHub

I am testing with 0.19.0.RC2.

--
Matt Weber
Sent with Sparrow

mattweber · February 21, 2012, 8:23pm

Thanks!

--
Matt Weber
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Tuesday, February 21, 2012 at 9:17 AM, Ivan Brusic wrote:

I have seen queries not returning if there are older versions of
Lucene in the classpath. Not every query uses every Lucene
class/method. Certain queries might use a different method, and if the
previous version of Lucene does not have that method, a query to ES
will simply hang with no exception thrown on the client or server
side.

--
Ivan

On Tue, Feb 21, 2012 at 8:29 AM, Shay Banon <kimchy@gmail.com (mailto:kimchy@gmail.com)> wrote:

It seems like its stuck in a look within Lucene boolean query, still need to
chase it more and figure out what goes wrong, can you open an issue just so
we won't loose it?

On Monday, February 20, 2012 at 5:19 PM, Matt Weber wrote:

The gist has the sample docs and sample query that locks it up. Just to
double check I installed a fresh copy 0.19.0.RC2, started it, copy and
pasted the test docs from the gist, then copy and pasted the query from the
gist and it locked up.

failed query (has filter) · GitHub

--
Matt Weber
Sent with Sparrow

On Monday, February 20, 2012 at 5:05 AM, Shay Banon wrote:

Regarding the recreation, I ran it against a simple index, and it doesn't
lock up, can you provide a recreation that shows the lock up starting fresh
(possibly indexing sample docs)?

Regarding boosting specific type, you can use the custom filters score query
(http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query.html),
and have a term filter on _type that boosts the score if the document
matches that type.

On Saturday, February 18, 2012 at 8:43 PM, Matt Weber wrote:

I have a need to make sure all hits from a specific field in a specific type
are always the highest ranked documents. To do this, I have used a constant
score query with a really high boost value against this type. At the same
time, I still want to search across other types and fields in my index. I
have crafted a boolean query with 2 should clauses. The first contains the
constant score query and the second contains a query string query.
Everything works as expected until my constant score query uses a filter.
When I specify a filter, the query locks up and never returns a response.

The only reason I decided to use a filter was for better performance. If I
remove the 2nd clause from the boolean query, the search executes just fine.
If I leave the 2nd clause, and remove the filter from the constant score
query, it also returns fine. If I do the constant score filter query by
itself, it also returns the expected query.

Here is a gist with the simplest example of the problem:
failed query (has filter) · GitHub

I am testing with 0.19.0.RC2.

--
Matt Weber
Sent with Sparrow