Query hits total greater than number of results returned

Elastic Stack Elasticsearch
1

I'm indexing 100 documents, then (after 2 seconds) I'm running a query that
based on the data in a particular field. The result.hits.total says 100 but
often there are only ~70 results returned. I've tried expicitly refreshing
the index after adding the docs but it doesn't seem to make any difference.

If I wait a minute or two I do get the output I'm expecting and I
understand there can be an indexing delay however saying there are 100
results but only returning 70 seems like a bug to me.

--

2

Are you setting your ID for your documents? Is it possible that sometimes you
update an existing document instead of creating it?
Do you use Bulk indexing? If so, did you check bulkResponses ?

Do you have always the same amount of documents when you search for it?
What query do you run? Did you try a matchAll() to see if all docs are here?

David

Le 16 janvier 2013 à 12:37, random35743373@googlemail.com a écrit :

I'm indexing 100 documents, then (after 2 seconds) I'm running a query that
based on the data in a particular field. The result.hits.total says 100 but
often there are only ~70 results returned. I've tried expicitly refreshing the
index after adding the docs but it doesn't seem to make any difference.

If I wait a minute or two I do get the output I'm expecting and I understand
there can be an indexing delay however saying there are 100 results but only
returning 70 seems like a bug to me.

--

--

3

I've just realised the title may be erroneous, I meant querying in the
generic sense, I'm actually running a search (see exact call below).

I'm setting an ID for each doc. Before I index the docs I delete the index
so I'm starting with a clean slate. I'm not using bulk anything. Every time
I query the hits total says 100 (which is correct).

Here is my search
http://127.0.01:9200/my_index/my_doc/_search
{
"fields": [
"my_field"
],
"size": 200,
"from": 0,
"query": {
"term": {
"my_field": "my_val"
}
}
}

On Wednesday, January 16, 2013 11:44:52 AM UTC, David Pilato wrote:

Are you setting your ID for your documents? Is it possible that
sometimes you update an existing document instead of creating it?
Do you use Bulk indexing? If so, did you check bulkResponses ?

Do you have always the same amount of documents when you search for it?
What query do you run? Did you try a matchAll() to see if all docs are
here?

David

Le 16 janvier 2013 à 12:37, random3...@googlemail.com <javascript:> a
écrit :

I'm indexing 100 documents, then (after 2 seconds) I'm running a query
that based on the data in a particular field. The result.hits.total says
100 but often there are only ~70 results returned. I've tried expicitly
refreshing the index after adding the docs but it doesn't seem to make any
difference.

If I wait a minute or two I do get the output I'm expecting and I
understand there can be an indexing delay however saying there are 100
results but only returning 70 seems like a bug to me.

--

--

4

I've just realised the title may be erroneous, I meant querying in the
generic sense, I'm actually running a search (see exact call below).

I'm setting an ID for each doc. Before I index the docs I delete the index
so I'm starting with a clean slate. I'm not using bulk anything. Every time
I query the hits total says 100 (which is correct).

Here is my search
http://127.0.01:9200/my_index/my_doc/_search
#{

"fields": [

"my_field"

],

"size": 200,

"from": 0,

"query": {

"term": {

"my_field": "my_val"

}

}

#}

On Wednesday, January 16, 2013 11:44:52 AM UTC, David Pilato wrote:

Are you setting your ID for your documents? Is it possible that
sometimes you update an existing document instead of creating it?
Do you use Bulk indexing? If so, did you check bulkResponses ?

Do you have always the same amount of documents when you search for it?
What query do you run? Did you try a matchAll() to see if all docs are
here?

David

Le 16 janvier 2013 à 12:37, random3...@googlemail.com <javascript:> a
écrit :

I'm indexing 100 documents, then (after 2 seconds) I'm running a query
that based on the data in a particular field. The result.hits.total says
100 but often there are only ~70 results returned. I've tried expicitly
refreshing the index after adding the docs but it doesn't seem to make any
difference.

If I wait a minute or two I do get the output I'm expecting and I
understand there can be an indexing delay however saying there are 100
results but only returning 70 seems like a bug to me.

--

--

5

Edit: I've posted this twice already and it keep being deleted, what am I
doing wrong?

I've just realised the title may be erroneous, I meant querying in the
generic sense, I'm actually running a search (see exact call below).

I'm setting an ID for each doc. Before I index the docs I delete the index
so I'm starting with a clean slate. I'm not using bulk anything. Every time
I query the hits total says 100 (which is correct).

Here is my search
/my_index/my_doc/_search
_ {
_ "fields": [
_ "my_field"
_ ],
_ "size": 200,
_ "from": 0,
_ "query": {
_ "term": {
_ "my_field": "my_val"
_ }
_ }
_ }

On Wednesday, January 16, 2013 11:44:52 AM UTC, David Pilato wrote:

Are you setting your ID for your documents? Is it possible that
sometimes you update an existing document instead of creating it?
Do you use Bulk indexing? If so, did you check bulkResponses ?

Do you have always the same amount of documents when you search for it?
What query do you run? Did you try a matchAll() to see if all docs are
here?

David

Le 16 janvier 2013 à 12:37, random3...@googlemail.com <javascript:> a
écrit :

I'm indexing 100 documents, then (after 2 seconds) I'm running a query
that based on the data in a particular field. The result.hits.total says
100 but often there are only ~70 results returned. I've tried expicitly
refreshing the index after adding the docs but it doesn't seem to make any
difference.

If I wait a minute or two I do get the output I'm expecting and I
understand there can be an indexing delay however saying there are 100
results but only returning 70 seems like a bug to me.

--

--

6

Are you able to build a curl recreation and gist it to help to reproduce your
use case?

Le 16 janvier 2013 à 13:08, random35743373@googlemail.com a écrit :

Edit: I've posted this twice already and it keep being deleted, what am I
doing wrong?

I've just realised the title may be erroneous, I meant querying in the
generic sense, I'm actually running a search (see exact call below).

I'm setting an ID for each doc. Before I index the docs I delete the index so
I'm starting with a clean slate. I'm not using bulk anything. Every time I
query the hits total says 100 (which is correct).

Here is my search
/my_index/my_doc/_search
_ {
_ "fields": [
_ "my_field"
_ ],
_ "size": 200,
_ "from": 0,
_ "query": {
_ "term": {
_ "my_field": "my_val"
_ }
_ }
_ }

On Wednesday, January 16, 2013 11:44:52 AM UTC, David Pilato wrote:

Are you setting your ID for your documents? Is it possible that
sometimes you update an existing document instead of creating it?
Do you use Bulk indexing? If so, did you check bulkResponses ?

Do you have always the same amount of documents when you search for it?
What query do you run? Did you try a matchAll() to see if all docs are
here?

David

Le 16 janvier 2013 à 12:37, random3...@googlemail.com a écrit :

> > > I'm indexing 100 documents, then (after 2 seconds) I'm running a
> > > query that based on the data in a particular field. The
> > > result.hits.total says 100 but often there are only ~70 results
> > > returned. I've tried expicitly refreshing the index after adding
> > > the docs but it doesn't seem to make any difference.

If I wait a minute or two I do get the output I'm expecting and I

understand there can be an indexing delay however saying there are 100
results but only returning 70 seems like a bug to me.

--

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--