Filtering docs on boolean value

kikster · September 11, 2011, 1:55am

I've been trying to get these types of queries working for a while,
but no luck. I'm trying to filter documents based on a boolean value
(basically only show documents where show == true).

Here's my query:

gist.github.com

https://gist.github.com/cdolivares/1158523

gistfile1.txt

{
"query" : {
 "filtered" : {
   "query" : {
     "custom_score" : {
       "query" : {
         "nested" : {
           "path" : "path1", 
           "query" : {
             "flt" : {

This file has been truncated. show original

Documents are being matched where show == false. Any ideas why this
would happen even though I explicitly say to filter them out?

kimchy · September 11, 2011, 10:49am

This should work. Can you post a recreation (no need for the custom score
query you use, just use match_all). See Elasticsearch Platform — Find real-time answers at scale | Elastic

On Sun, Sep 11, 2011 at 4:55 AM, Chris
christopherdavidolivares@gmail.comwrote:

I've been trying to get these types of queries working for a while,
but no luck. I'm trying to filter documents based on a boolean value
(basically only show documents where show == true).

Here's my query:

1158523’s gists · GitHub

Documents are being matched where show == false. Any ideas why this
would happen even though I explicitly say to filter them out?

kikster · September 11, 2011, 9:18pm

I can't seem to reproduce the error with curl recreations. When I run
the query against my dataset, there are a lot of offending documents.
However, when I created a new index with the data of a few of the
offending documents, everything works great. The documents that
wrongly match the query in my original dataset are correctly filtered
when I create the new index (even though I only indexed 3 documents to
test). I'm completely stumped. I have around 600 indexed documents
(each one around 20kB, and am running elasticsearch on 64Mb of memory,
far below the recommended standard configuration. Could malformed
data in a few of my documents cause the query to not match? Could
running on 64Mb of memory be the issue?

On Sep 11, 3:49 am, Shay Banon kim...@gmail.com wrote:

This should work. Can you post a recreation (no need for the custom score
query you use, just use match_all). Seehttp://www.elasticsearch.org/help

On Sun, Sep 11, 2011 at 4:55 AM, Chris
christopherdavidoliva...@gmail.comwrote:

I've been trying to get these types of queries working for a while,
but no luck. I'm trying to filter documents based on a boolean value
(basically only show documents where show == true).

Here's my query:

1158523’s gists · GitHub

Documents are being matched where show == false. Any ideas why this
would happen even though I explicitly say to filter them out?

kikster · September 12, 2011, 2:09am

Finally able to reproduce the error.

gist.github.com

https://gist.github.com/cdolivares/1210453

gistfile1.json

curl -XPUT 'http://localhost:9200/testings/' -d '{
    "index" : {
        "number_of_shards" : 1
    }
    
}'

curl -XPUT 'http://localhost:9200/testings/testing/_mapping' -d '{
    "testing" : {
        "properties" : {

This file has been truncated. show original

On Sep 11, 2:18 pm, Chris christopherdavidoliva...@gmail.com wrote:

I can't seem to reproduce the error with curl recreations. When I run
the query against my dataset, there are a lot of offending documents.
However, when I created a new index with the data of a few of the
offending documents, everything works great. The documents that
wrongly match the query in my original dataset are correctly filtered
when I create the new index (even though I only indexed 3 documents to
test). I'm completely stumped. I have around 600 indexed documents
(each one around 20kB, and am running elasticsearch on 64Mb of memory,
far below the recommended standard configuration. Could malformed
data in a few of my documents cause the query to not match? Could
running on 64Mb of memory be the issue?

On Sep 11, 3:49 am, Shay Banon kim...@gmail.com wrote:

This should work. Can you post a recreation (no need for the custom score
query you use, just use match_all). Seehttp://www.elasticsearch.org/help

On Sun, Sep 11, 2011 at 4:55 AM, Chris
christopherdavidoliva...@gmail.comwrote:

I've been trying to get these types of queries working for a while,
but no luck. I'm trying to filter documents based on a boolean value
(basically only show documents where show == true).

Here's my query:

1158523’s gists · GitHub

Documents are being matched where show == false. Any ideas why this
would happen even though I explicitly say to filter them out?

kikster · September 12, 2011, 2:30am

One more thing, the filter works properly when I index the data
without setting "reviews" to type "nested".

On Sep 11, 7:09 pm, Chris christopherdavidoliva...@gmail.com wrote:

Finally able to reproduce the error.

gist:1210453 · GitHub

On Sep 11, 2:18 pm, Chris christopherdavidoliva...@gmail.com wrote:

I can't seem to reproduce the error with curl recreations. When I run
the query against my dataset, there are a lot of offending documents.
However, when I created a new index with the data of a few of the
offending documents, everything works great. The documents that
wrongly match the query in my original dataset are correctly filtered
when I create the new index (even though I only indexed 3 documents to
test). I'm completely stumped. I have around 600 indexed documents
(each one around 20kB, and am running elasticsearch on 64Mb of memory,
far below the recommended standard configuration. Could malformed
data in a few of my documents cause the query to not match? Could
running on 64Mb of memory be the issue?

On Sep 11, 3:49 am, Shay Banon kim...@gmail.com wrote:

This should work. Can you post a recreation (no need for the custom score
query you use, just use match_all). Seehttp://www.elasticsearch.org/help

On Sun, Sep 11, 2011 at 4:55 AM, Chris
christopherdavidoliva...@gmail.comwrote:

I've been trying to get these types of queries working for a while,
but no luck. I'm trying to filter documents based on a boolean value
(basically only show documents where show == true).

Here's my query:

1158523’s gists · GitHub

Documents are being matched where show == false. Any ideas why this
would happen even though I explicitly say to filter them out?

kimchy · September 12, 2011, 8:26am

Hi,

Thanks for the recreation!. Found the problem (a tricky one to track...),
opened an issue: Nested Mapping: Nested object with a null value causes wrong indexing structure (resulting in wrong search responses) · Issue #1323 · elastic/elasticsearch · GitHub.
Fixed in 0.17 branch and master. 0.17.7 will be released in a day or two,
once Lucene 3.4.0 is out.

-shay.banon

On Mon, Sep 12, 2011 at 5:30 AM, Chris
christopherdavidolivares@gmail.comwrote:

One more thing, the filter works properly when I index the data
without setting "reviews" to type "nested".

On Sep 11, 7:09 pm, Chris christopherdavidoliva...@gmail.com wrote:

Finally able to reproduce the error.

gist:1210453 · GitHub

On Sep 11, 2:18 pm, Chris christopherdavidoliva...@gmail.com wrote:

I can't seem to reproduce the error with curl recreations. When I run
the query against my dataset, there are a lot of offending documents.
However, when I created a new index with the data of a few of the
offending documents, everything works great. The documents that
wrongly match the query in my original dataset are correctly filtered
when I create the new index (even though I only indexed 3 documents to
test). I'm completely stumped. I have around 600 indexed documents
(each one around 20kB, and am running elasticsearch on 64Mb of memory,
far below the recommended standard configuration. Could malformed
data in a few of my documents cause the query to not match? Could
running on 64Mb of memory be the issue?

On Sep 11, 3:49 am, Shay Banon kim...@gmail.com wrote:

This should work. Can you post a recreation (no need for the custom
score
query you use, just use match_all). Seehttp://
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Sun, Sep 11, 2011 at 4:55 AM, Chris
christopherdavidoliva...@gmail.comwrote:

I've been trying to get these types of queries working for a while,
but no luck. I'm trying to filter documents based on a boolean
value
(basically only show documents where show == true).

Here's my query:

1158523’s gists · GitHub

Documents are being matched where show == false. Any ideas why
this
would happen even though I explicitly say to filter them out?