Missing filter on fields which are nested objects

Hi

The "missing" filter doesn't seem to work on fields which are nested
objects. Its supposed to return documents which don't have the specified
field, and it works for "flattened" attributes. But if the field itself
is a nested object, it doesn't work. Though it works again on attributes
on the nested object itself.

Is this a bug or expected behavior? How can i find documents which are
missing some fields which are nested objects?

To illustrate (https://gist.github.com/abhi-sanoujam/5720015):

  • Put 3 documents
  • Search for "missing" a field which is top level attribute - WORKS
  • Search for "missing" a field which is a nested object - DOESN'T WORK
  • Search for "missing" a field within a nested object - WORKS

curl -XPUT http://localhost:9200/test/test/1 -d '{
"a" : "A1",
"b" : {
"b_a" : "b_a 1"
}
}'

curl -XPUT http://localhost:9200/test/test/2 -d '{
"abc" : "abc 1"
}'

curl -XPUT http://localhost:9200/test/test/3 -d '{
"a" : "A3",
"b" : {
"b_b" : "b_b 1"
}
}'

curl -XPOST "http://localhost:9200/test/_search?pretty"
Returns all docs*
*
1) Search for missing field which is not a nested object - WORKS
curl -XPOST "http://localhost:9200/test/_search?pretty" -d '{
"filter" : {
"missing" : {
"field" : "a"
}
}
}'
Returns document with id=2 (which has only field "abc") [EXPECTED]

2) Search for missing field which is a nested object - DOESN'T WORK
curl -XPOST "http://localhost:9200/test/_search?pretty" -d '{
"filter" : {
"missing" : {
"field" : "b"
}
}
}'

Returns all docs. Expected to have returned only doc with id=2, as
that's the only document that's missing the attribute "b".
[NOT EXPECTED]

3) Search for missing field within nested objects - WORKS
curl -XPOST "http://localhost:9200/test/_search?pretty" -d '{
"filter" : {
"missing" : {
"field" : "b.b_a"
}
}
}'
Returns doc with id 2 and 3 [EXPECTED]

curl -XPOST "http://localhost:9200/test/_search?pretty" -d '{
"filter" : {
"missing" : {
"field" : "b.b_b"
}
}
}'
*Returns doc with id 1 and 2 *[EXPECTED]

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey,

currently it is not possible to apply the missing filter on an object
(because the object is not really indexed compared to the fields inside of
the object).
The current workaround would be to call missing on a field inside of the
object.
As this is not really userfriendly, I have created an issue for that.

--Alex

On Thu, Jun 6, 2013 at 10:10 AM, Abhishek Sanoujam
abhi.sanoujam@gmail.comwrote:

Hi

The "missing" filter doesn't seem to work on fields which are nested
objects. Its supposed to return documents which don't have the specified
field, and it works for "flattened" attributes. But if the field itself is
a nested object, it doesn't work. Though it works again on attributes on
the nested object itself.

Is this a bug or expected behavior? How can i find documents which are
missing some fields which are nested objects?

To illustrate (doesnt_work.sh · GitHub):

  • Put 3 documents
  • Search for "missing" a field which is top level attribute - WORKS
  • Search for "missing" a field which is a nested object - DOESN'T WORK
  • Search for "missing" a field within a nested object - WORKS

curl -XPUT http://localhost:9200/test/test/1 -d '{
"a" : "A1",
"b" : {
"b_a" : "b_a 1"
}
}'

curl -XPUT http://localhost:9200/test/test/2 -d '{
"abc" : "abc 1"
}'

curl -XPUT http://localhost:9200/test/test/3 -d '{
"a" : "A3",
"b" : {
"b_b" : "b_b 1"
}
}'

curl -XPOST "http://localhost:9200/test/_search?pretty"http://localhost:9200/test/_search?pretty
Returns all docs*
*
1) Search for missing field which is not a nested object - WORKS
curl -XPOST "http://localhost:9200/test/_search?pretty"http://localhost:9200/test/_search?pretty-d '{
"filter" : {
"missing" : {
"field" : "a"
}
}
}'
Returns document with id=2 (which has only field "abc") [EXPECTED]

2) Search for missing field which is a nested object - DOESN'T WORK
curl -XPOST "http://localhost:9200/test/_search?pretty"http://localhost:9200/test/_search?pretty-d '{
"filter" : {
"missing" : {
"field" : "b"
}
}
}'

Returns all docs. Expected to have returned only doc with id=2, as
that's the only document that's missing the attribute "b".
[NOT EXPECTED]

3) Search for missing field within nested objects - WORKS
curl -XPOST "http://localhost:9200/test/_search?pretty"http://localhost:9200/test/_search?pretty-d '{
"filter" : {
"missing" : {
"field" : "b.b_a"
}
}
}'
Returns doc with id 2 and 3 [EXPECTED]

curl -XPOST "http://localhost:9200/test/_search?pretty"http://localhost:9200/test/_search?pretty-d '{
"filter" : {
"missing" : {
"field" : "b.b_b"
}
}
}'
*Returns doc with id 1 and 2 *[EXPECTED]

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for the input Alex. Hope the issue gets fixed pretty soon :slight_smile:
For my use case, I don't really have a choice as the fields inside the
object are also dynamic and there's no fixed "attribute" in the nested
object that I can use to decide the existence.

Looks like i'll have to workaround and update the document by adding a
field to mark existence/non-existence of certain nested objects.

On 6/6/13 2:25 PM, Alexander Reelsen wrote:

Hey,

currently it is not possible to apply the missing filter on an object
(because the object is not really indexed compared to the fields
inside of the object).
The current workaround would be to call missing on a field inside of
the object.
As this is not really userfriendly, I have created an issue for that.

missing/exists filters should also work for objects · Issue #3141 · elastic/elasticsearch · GitHub

--Alex

On Thu, Jun 6, 2013 at 10:10 AM, Abhishek Sanoujam
<abhi.sanoujam@gmail.com mailto:abhi.sanoujam@gmail.com> wrote:

Hi

The "missing" filter doesn't seem to work on fields which are
nested objects. Its supposed to return documents which don't have
the specified field, and it works for "flattened" attributes. But
if the field itself is a nested object, it doesn't work. Though it
works again on attributes on the nested object itself.

Is this a bug or expected behavior? How can i find documents which
are missing some fields which are nested objects?

To illustrate (https://gist.github.com/abhi-sanoujam/5720015):
- Put 3 documents
- Search for "missing" a field which is top level attribute - WORKS
- Search for "missing" a field which is a nested object - DOESN'T WORK
- Search for "missing" a field within a nested object - WORKS

curl -XPUT http://localhost:9200/test/test/1 -d '{
    "a" : "A1",
    "b" : {
        "b_a" : "b_a 1"
    }
}'

curl -XPUT http://localhost:9200/test/test/2 -d '{
    "abc" : "abc 1"
}'

curl -XPUT http://localhost:9200/test/test/3 -d '{
    "a" : "A3",
    "b" : {
        "b_b" : "b_b 1"
    }
}'

curl -XPOST "http://localhost:9200/test/_search?pretty"
<http://localhost:9200/test/_search?pretty>
Returns all docs*
*
*1) Search for missing field which is not a nested object - WORKS*
curl -XPOST "http://localhost:9200/test/_search?pretty"
<http://localhost:9200/test/_search?pretty> -d '{
    "filter" : {
        "missing" : {
            "field" : "a"
        }
    }
}'
*Returns document with id=2 (which has only field "abc") [EXPECTED]*

*2) Search for missing field which is a nested object - DOESN'T WORK*
curl -XPOST "http://localhost:9200/test/_search?pretty"
<http://localhost:9200/test/_search?pretty> -d '{
    "filter" : {
        "missing" : {
            "field" : "b"
        }
    }
}'

*Returns all docs. **Expected to have returned only doc with
id=2**, as that's the only document that's missing the attribute
"b".* [NOT EXPECTED]

*3) Search for missing field within nested objects - WORKS*
curl -XPOST "http://localhost:9200/test/_search?pretty"
<http://localhost:9200/test/_search?pretty> -d '{
    "filter" : {
        "missing" : {
            "field" : "b.b_a"
        }
    }
}'
*Returns doc with id 2 and 3 [EXPECTED]*

curl -XPOST "http://localhost:9200/test/_search?pretty"
<http://localhost:9200/test/_search?pretty> -d '{
    "filter" : {
        "missing" : {
            "field" : "b.b_b"
        }
    }
}'
*Returns doc with id 1 and 2 **[EXPECTED]*

-- 
---------
Cheers,
Abhishek

-- 
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,
Abhishek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.