Ignore non-existent docs in mget

Hi, ES noob here, bear with me.

I need to filter out the results from an mget so that those with
exists=false are not returned. How can I do this?

The docs state:
"All multi indices API support the ignore_indices option. Setting it to
missing will cause indices that do not exists to be ignored from the
execution..."

But I can't seem to get this to work. I've tried something like this (and
various combinations), to no avail:

]$ curl -XGET 'http://localhost:9200/idx/type/_mget?ignore_indices=missing'
-d '{"ids": ["xxx"]}'

Thanks,
Allan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Allan Johns wrote:

I need to filter out the results from an mget so that those with
exists=false are not returned. How can I do this?

[...]

But I can't seem to get this to work. I've tried something like
this (and various combinations), to no avail:

]$ curl -XGET
'http://localhost:9200/idx/type/_mget?ignore_indices=missing'
-d '{"ids": ["xxx"]}'

ignore_indices is only useful when you're using an index wildcard.
For example:

curl -s
'localhost:9200/foo*/_search?ignore_indices=missing&size=1'

If no index matches the foo* pattern, you won't get an error.
This would have the same effect as searching the entire cluster.

In your case, you're not using a wildcard for the index name, but
more importantly, you're using mget which is only useful when you
know exactly the index/type/id triplet that will reference the
docs you want.

Back to your original need, what do you mean by needing "those
with exists=false" to be omitted? You have a field called
"exists" and want to search where it has a true value?

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I want those nonexistent documents not to be returned in the result set at
all. So: I know the index, type and set of IDs, however some IDs will not
exist in the database, and I don't know this ahead of time. I simply want
those nonexistent documents to be silently removed from the result set. Is
this possible?

"Back to your original need, what do you mean by needing "those with
exists=false" to be omitted? You have a field called "exists" and want to
search where it has a true value?"

Consider:

$ curl -XGET 'http://localhost:9200/idx/type/SOME_BAD_ID'
{"_index":"fury","_type":"products","_id":"ID0","exists":false}

I want a result set that does not include these exists=false docs at all.

thanks
Allan

On Wed, May 22, 2013 at 1:40 AM, Drew Raines aaraines@gmail.com wrote:

Allan Johns wrote:

I need to filter out the results from an mget so that those with

exists=false are not returned. How can I do this?

[...]

But I can't seem to get this to work. I've tried something like this (and

various combinations), to no avail:

]$ curl -XGET 'http://localhost:9200/idx/**type/_mget?ignore_indices=**
missing http://localhost:9200/idx/type/_mget?ignore_indices=missing'
-d '{"ids": ["xxx"]}'

ignore_indices is only useful when you're using an index wildcard. For
example:

curl -s 'localhost:9200/foo*/_search?**ignore_indices=missing&size=1'

If no index matches the foo* pattern, you won't get an error. This would
have the same effect as searching the entire cluster.

In your case, you're not using a wildcard for the index name, but more
importantly, you're using mget which is only useful when you know exactly
the index/type/id triplet that will reference the docs you want.

Back to your original need, what do you mean by needing "those with
exists=false" to be omitted? You have a field called "exists" and want to
search where it has a true value?

Drew

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sorry I meant to show that example using mget:

]$ curl -XGET 'http://localhost:9200/fury/products/_mget' -d '{"ids":
["BAD_ID","GOOD_ID"]}'
{"docs":[{"_index":"fury","_type":"products","_id":"BAD_ID","exists":false},{"_index":"fury","_type":"products","_id":"GOOD_ID","foo":"bah","etc":"etcetc"}]}

What I want is this behaviour:

]$ curl -XGET 'http://localhost:9200/fury/products/_mget' -d '{"ids":
["BAD_ID", "GOOD_ID"]}'
{"docs":[{"_index":"fury","_type":"products","_id":"GOOD_ID","foo":"bah","etc":"etcetc"}]}

On Wed, May 29, 2013 at 12:19 PM, Allan Johns nerdvegas@gmail.com wrote:

I want those nonexistent documents not to be returned in the result set at
all. So: I know the index, type and set of IDs, however some IDs will not
exist in the database, and I don't know this ahead of time. I simply want
those nonexistent documents to be silently removed from the result set. Is
this possible?

"Back to your original need, what do you mean by needing "those with
exists=false" to be omitted? You have a field called "exists" and want to
search where it has a true value?"

Consider:

$ curl -XGET 'http://localhost:9200/idx/type/SOME_BAD_ID'
{"_index":"fury","_type":"products","_id":"ID0","exists":false}

I want a result set that does not include these exists=false docs at all.

thanks
Allan

On Wed, May 22, 2013 at 1:40 AM, Drew Raines aaraines@gmail.com wrote:

Allan Johns wrote:

I need to filter out the results from an mget so that those with

exists=false are not returned. How can I do this?

[...]

But I can't seem to get this to work. I've tried something like this

(and various combinations), to no avail:

]$ curl -XGET 'http://localhost:9200/idx/**type/_mget?ignore_indices=**
missing http://localhost:9200/idx/type/_mget?ignore_indices=missing'
-d '{"ids": ["xxx"]}'

ignore_indices is only useful when you're using an index wildcard. For
example:

curl -s 'localhost:9200/foo*/_search?**ignore_indices=missing&size=1'

If no index matches the foo* pattern, you won't get an error. This would
have the same effect as searching the entire cluster.

In your case, you're not using a wildcard for the index name, but more
importantly, you're using mget which is only useful when you know exactly
the index/type/id triplet that will reference the docs you want.

Back to your original need, what do you mean by needing "those with
exists=false" to be omitted? You have a field called "exists" and want to
search where it has a true value?

Drew

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You can easily filter that on client side.
Is your concern about network usage?

May be you should open an issue to have an 'ignore_missing' option for _mget API (default to false)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2013 à 04:22, Allan Johns nerdvegas@gmail.com a écrit :

Sorry I meant to show that example using mget:

]$ curl -XGET 'http://localhost:9200/fury/products/_mget' -d '{"ids": ["BAD_ID","GOOD_ID"]}'
{"docs":[{"_index":"fury","_type":"products","_id":"BAD_ID","exists":false},{"_index":"fury","_type":"products","_id":"GOOD_ID","foo":"bah","etc":"etcetc"}]}

What I want is this behaviour:

]$ curl -XGET 'http://localhost:9200/fury/products/_mget' -d '{"ids": ["BAD_ID", "GOOD_ID"]}'
{"docs":[{"_index":"fury","_type":"products","_id":"GOOD_ID","foo":"bah","etc":"etcetc"}]}

On Wed, May 29, 2013 at 12:19 PM, Allan Johns nerdvegas@gmail.com wrote:

I want those nonexistent documents not to be returned in the result set at all. So: I know the index, type and set of IDs, however some IDs will not exist in the database, and I don't know this ahead of time. I simply want those nonexistent documents to be silently removed from the result set. Is this possible?

"Back to your original need, what do you mean by needing "those with exists=false" to be omitted? You have a field called "exists" and want to search where it has a true value?"

Consider:

$ curl -XGET 'http://localhost:9200/idx/type/SOME_BAD_ID'
{"_index":"fury","_type":"products","_id":"ID0","exists":false}

I want a result set that does not include these exists=false docs at all.

thanks
Allan

On Wed, May 22, 2013 at 1:40 AM, Drew Raines aaraines@gmail.com wrote:

Allan Johns wrote:

I need to filter out the results from an mget so that those with exists=false are not returned. How can I do this?

[...]

But I can't seem to get this to work. I've tried something like this (and various combinations), to no avail:

]$ curl -XGET 'http://localhost:9200/idx/type/_mget?ignore_indices=missing'
-d '{"ids": ["xxx"]}'

ignore_indices is only useful when you're using an index wildcard. For example:

curl -s 'localhost:9200/foo*/_search?ignore_indices=missing&size=1'

If no index matches the foo* pattern, you won't get an error. This would have the same effect as searching the entire cluster.

In your case, you're not using a wildcard for the index name, but more importantly, you're using mget which is only useful when you know exactly the index/type/id triplet that will reference the docs you want.

Back to your original need, what do you mean by needing "those with exists=false" to be omitted? You have a field called "exists" and want to search where it has a true value?

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

My concern isn't network usage, it's paging. Sometimes my result set is
quite large, and I want to only page into the results when necessary. If
missing docs are included in the response, I'm forced to fetch the whole
lot, in order to filter out missing docs client-side, so that I'm reporting
the correct document count to the client app.

If I can't do this currently and others agree this is a reasonable thing to
want to do, I'll open an issue.

thx
A

On Wed, May 29, 2013 at 5:39 PM, David Pilato david@pilato.fr wrote:

You can easily filter that on client side.
Is your concern about network usage?

May be you should open an issue to have an 'ignore_missing' option for
_mget API (default to false)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2013 à 04:22, Allan Johns nerdvegas@gmail.com a écrit :

Sorry I meant to show that example using mget:

]$ curl -XGET 'http://localhost:9200/fury/products/_mget' -d '{"ids":
["BAD_ID","GOOD_ID"]}'

{"docs":[{"_index":"fury","_type":"products","_id":"BAD_ID","exists":false},{"_index":"fury","_type":"products","_id":"GOOD_ID","foo":"bah","etc":"etcetc"}]}

What I want is this behaviour:

]$ curl -XGET 'http://localhost:9200/fury/products/_mget' -d '{"ids":
["BAD_ID", "GOOD_ID"]}'

{"docs":[{"_index":"fury","_type":"products","_id":"GOOD_ID","foo":"bah","etc":"etcetc"}]}

On Wed, May 29, 2013 at 12:19 PM, Allan Johns nerdvegas@gmail.com wrote:

I want those nonexistent documents not to be returned in the result set
at all. So: I know the index, type and set of IDs, however some IDs will
not exist in the database, and I don't know this ahead of time. I simply
want those nonexistent documents to be silently removed from the result
set. Is this possible?

"Back to your original need, what do you mean by needing "those with
exists=false" to be omitted? You have a field called "exists" and want to
search where it has a true value?"

Consider:

$ curl -XGET 'http://localhost:9200/idx/type/SOME_BAD_ID'
{"_index":"fury","_type":"products","_id":"ID0","exists":false}

I want a result set that does not include these exists=false docs at all.

thanks
Allan

On Wed, May 22, 2013 at 1:40 AM, Drew Raines aaraines@gmail.com wrote:

Allan Johns wrote:

I need to filter out the results from an mget so that those with

exists=false are not returned. How can I do this?

[...]

But I can't seem to get this to work. I've tried something like this

(and various combinations), to no avail:

]$ curl -XGET 'http://localhost:9200/idx/**type/_mget?ignore_indices=**
missing http://localhost:9200/idx/type/_mget?ignore_indices=missing'
-d '{"ids": ["xxx"]}'

ignore_indices is only useful when you're using an index wildcard. For
example:

curl -s 'localhost:9200/foo*/_search?**ignore_indices=missing&size=1'

If no index matches the foo* pattern, you won't get an error. This would
have the same effect as searching the entire cluster.

In your case, you're not using a wildcard for the index name, but more
importantly, you're using mget which is only useful when you know exactly
the index/type/id triplet that will reference the docs you want.

Back to your original need, what do you mean by needing "those with
exists=false" to be omitted? You have a field called "exists" and want to
search where it has a true value?

Drew

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.