# Bulk indexing and count mismatch
**URL:** https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107
**Category:** Elasticsearch
**Created:** [June 14, 2012, 4:40pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107 "2012-06-14T16:40:33Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@phoenix](https://discuss.elastic.co/u/phoenix)
#### Post date: [June 14, 2012, 4:40pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/1 "2012-06-14T16:40:33Z")
Hi all,
I'm facing a strange situation.
I'm indexing some documents using the bulk java api.
My problem is in my tests. First my test uses the bulk api to index 100
randomly created documents (with random generated strings). Then it
executes a count query to check i now have a 100 documents in my index.
But actually i have 0, or maybe one. And sometime while debugging a
actually have 100 docs.
Is this a sycnhronization problem? I thought that the actionGet() method
was waiting for the job to be done before returning.
My code is as follows :
BulkResponse response = bulkRequest.execute().actionGet();
Any idea ?
Frederic
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [June 14, 2012, 5:15pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/2 "2012-06-14T17:15:07Z")
Are you indexing to a new id for each document? That might account for
seeing only one document, but zero documents is a different issue.
--
Ivan
On Thu, Jun 14, 2012 at 9:40 AM, Frederic Esnault
[esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) wrote:
> Hi all,
>
> I'm facing a strange situation.
> I'm indexing some documents using the bulk java api.
> My problem is in my tests. First my test uses the bulk api to index 100
> randomly created documents (with random generated strings). Then it executes
> a count query to check i now have a 100 documents in my index.
> But actually i have 0, or maybe one. And sometime while debugging a actually
> have 100 docs.
> Is this a sycnhronization problem? I thought that the actionGet() method was
> waiting for the job to be done before returning.
>
> My code is as follows :
>
> BulkResponse response = bulkRequest.execute().actionGet();
>
> Any idea ?
>
> Frederic
---
### Author:  [@phoenix](https://discuss.elastic.co/u/phoenix)
#### Post date: [June 15, 2012, 7:16am UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/3 "2012-06-15T07:16:09Z")
We are not giving any id to our documents. According to the API,
Elasticsearch is supposed to generate one by itself.
And actually if we add a Thread.sleep(...) before asking for the document
count, we get the expected result. So it seems it just takes time.
But i was thinking (and reading the javadoc seems to confirm it) that the
execute().actionGet() call was waiting for the completion of the task to
return.
Is it different for bulk requests?
Is there any way to actually wait for the indexing to be done? (Not really
important at runtime, but for testing purposes, it is).
Frederic
On Thu, Jun 14, 2012 at 7:15 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> Are you indexing to a new id for each document? That might account for
> seeing only one document, but zero documents is a different issue.
>
> --
> Ivan
>
> On Thu, Jun 14, 2012 at 9:40 AM, Frederic Esnault
> [esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) wrote:
>
> > Hi all,
> >
> > I'm facing a strange situation.
> > I'm indexing some documents using the bulk java api.
> > My problem is in my tests. First my test uses the bulk api to index 100
> > randomly created documents (with random generated strings). Then it
> > executes
> > a count query to check i now have a 100 documents in my index.
> > But actually i have 0, or maybe one. And sometime while debugging a
> > actually
> > have 100 docs.
> > Is this a sycnhronization problem? I thought that the actionGet() method
> > was
> > waiting for the job to be done before returning.
> >
> > My code is as follows :
> >
> > BulkResponse response = bulkRequest.execute().actionGet();
> >
> > Any idea ?
> >
> > Frederic
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [June 15, 2012, 4:25pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/4 "2012-06-15T16:25:39Z")
The synchronous calls ensures that the operation is committed at the
server level, but there can still be delays at the Lucene level. The
default index refresh interval is 1 second. How many BulkItemResponses
do you have in your BulkResponse?
--
Ivan
On Fri, Jun 15, 2012 at 12:16 AM, Frederic Esnault
[esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) wrote:
> We are not giving any id to our documents. According to the API,
> Elasticsearch is supposed to generate one by itself.
> And actually if we add a Thread.sleep(...) before asking for the document
> count, we get the expected result. So it seems it just takes time.
> But i was thinking (and reading the javadoc seems to confirm it) that the
> execute().actionGet() call was waiting for the completion of the task to
> return.
> Is it different for bulk requests?
> Is there any way to actually wait for the indexing to be done? (Not really
> important at runtime, but for testing purposes, it is).
>
> Frederic
>
> On Thu, Jun 14, 2012 at 7:15 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > Are you indexing to a new id for each document? That might account for
> > seeing only one document, but zero documents is a different issue.
> >
> > --
> > Ivan
> >
> > On Thu, Jun 14, 2012 at 9:40 AM, Frederic Esnault
> > [esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) wrote:
> >
> > > Hi all,
> > >
> > > I'm facing a strange situation.
> > > I'm indexing some documents using the bulk java api.
> > > My problem is in my tests. First my test uses the bulk api to index 100
> > > randomly created documents (with random generated strings). Then it
> > > executes
> > > a count query to check i now have a 100 documents in my index.
> > > But actually i have 0, or maybe one. And sometime while debugging a
> > > actually
> > > have 100 docs.
> > > Is this a sycnhronization problem? I thought that the actionGet() method
> > > was
> > > waiting for the job to be done before returning.
> > >
> > > My code is as follows :
> > >
> > > BulkResponse response = bulkRequest.execute().actionGet();
> > >
> > > Any idea ?
> > >
> > > Frederic
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [June 15, 2012, 6:07pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/5 "2012-06-15T18:07:50Z")
Frederic,
Just add explicit refresh before checking document count:
client.admin().indices().prepareRefresh().execute().actionGet();
This command will ensure that all indexed records are committed and and
available in your searches.
On Friday, June 15, 2012 12:25:39 PM UTC-4, Ivan Brusic wrote:
> The synchronous calls ensures that the operation is committed at the
> server level, but there can still be delays at the Lucene level. The
> default index refresh interval is 1 second. How many BulkItemResponses
> do you have in your BulkResponse?
>
> --
> Ivan
>
> On Fri, Jun 15, 2012 at 12:16 AM, Frederic Esnault
> [esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) wrote:
>
> > We are not giving any id to our documents. According to the API,
> > Elasticsearch is supposed to generate one by itself.
> > And actually if we add a Thread.sleep(...) before asking for the
> > document
> > count, we get the expected result. So it seems it just takes time.
> > But i was thinking (and reading the javadoc seems to confirm it) that
> > the
> > execute().actionGet() call was waiting for the completion of the task to
> > return.
> > Is it different for bulk requests?
> > Is there any way to actually wait for the indexing to be done? (Not
> > really
> > important at runtime, but for testing purposes, it is).
> >
> > Frederic
> >
> > On Thu, Jun 14, 2012 at 7:15 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > Are you indexing to a new id for each document? That might account for
> > > seeing only one document, but zero documents is a different issue.
> > >
> > > --
> > > Ivan
> > >
> > > On Thu, Jun 14, 2012 at 9:40 AM, Frederic Esnault
> > > [esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm facing a strange situation.
> > > > I'm indexing some documents using the bulk java api.
> > > > My problem is in my tests. First my test uses the bulk api to index
> > > > 100
> > > > randomly created documents (with random generated strings). Then it
> > > > executes
> > > > a count query to check i now have a 100 documents in my index.
> > > > But actually i have 0, or maybe one. And sometime while debugging a
> > > > actually
> > > > have 100 docs.
> > > > Is this a sycnhronization problem? I thought that the actionGet()
> > > > method
> > > > was
> > > > waiting for the job to be done before returning.
> > > >
> > > > My code is as follows :
> > > >
> > > > BulkResponse response = bulkRequest.execute().actionGet();
> > > >
> > > > Any idea ?
> > > >
> > > > Frederic
---
### Author:  [@phoenix](https://discuss.elastic.co/u/phoenix)
#### Post date: [June 16, 2012, 5:20am UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/6 "2012-06-16T05:20:29Z")
@Ivan :
We have 1 or 100, depending on if we asked for sleep on the thread.
@Igor :
Thx for the tip, we'll try to refresh first, we'll let you know the result
🙂
Frederic
On Friday, June 15, 2012, Igor Motov wrote:
> Frederic,
>
> Just add explicit refresh before checking document count:
>
> client.admin().indices().prepareRefresh().execute().actionGet();
>
> This command will ensure that all indexed records are committed and and
> available in your searches.
>
> On Friday, June 15, 2012 12:25:39 PM UTC-4, Ivan Brusic wrote:
>
> > The synchronous calls ensures that the operation is committed at the
> > server level, but there can still be delays at the Lucene level. The
> > default index refresh interval is 1 second. How many BulkItemResponses
> > do you have in your BulkResponse?
> >
> > --
> > Ivan
> >
> > On Fri, Jun 15, 2012 at 12:16 AM, Frederic Esnault
> > \<[esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) \ > 'esnault.frederic@gmail.com');\>\> wrote:
> >
> > > We are not giving any id to our documents. According to the API,
> > > Elasticsearch is supposed to generate one by itself.
> > > And actually if we add a Thread.sleep(...) before asking for the
> > > document
> > > count, we get the expected result. So it seems it just takes time.
> > > But i was thinking (and reading the javadoc seems to confirm it) that
> > > the
> > > execute().actionGet() call was waiting for the completion of the task
> > > to
> > > return.
> > > Is it different for bulk requests?
> > > Is there any way to actually wait for the indexing to be done? (Not
> > > really
> > > important at runtime, but for testing purposes, it is).
> > >
> > > Frederic
> > >
> > > On Thu, Jun 14, 2012 at 7:15 PM, Ivan Brusic \<[ivan@brusic.com](mailto:ivan@brusic.com)\\>
> > > wrote:
> > >
> > > > Are you indexing to a new id for each document? That might account for
> > > > seeing only one document, but zero documents is a different issue.
> > > >
> > > > --
> > > > Ivan
> > > >
> > > > On Thu, Jun 14, 2012 at 9:40 AM, Frederic Esnault
> > > > \<[esnault.frederic@gmail.com](mailto:esnault.frederic@gmail.com) \ > > > 'esnault.frederic@gmail.com');\>\> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm facing a strange situation.
> > > > > I'm indexing some documents using the bulk java api.
> > > > > My problem is in my tests. First my test uses the bulk api to index
> > > > > 100
> > > > > randomly created documents (with random generated strings). Then it
> > > > > executes
> > > > > a count query to check i now have a 100 documents in my index.
> > > > > But actually i have 0, or maybe one. And sometime while debugging a
> > > > > actually
> > > > > have 100 docs.
> > > > > Is this a sycnhronization problem? I thought that the actionGet()
> > > > > method
> > > > > was
> > > > > waiting for the job to be done before returning.
> > > > >
> > > > > My code is as follows :
> > > > >
> > > > > BulkResponse response = bulkRequest.execute().\*\*actionGet();
> > > > >
> > > > > Any idea ?
> > > > >
> > > > > Frederic
---
### Author:  [@phoenix](https://discuss.elastic.co/u/phoenix)
#### Post date: [June 19, 2012, 7:46pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/7 "2012-06-19T19:46:32Z")
Thanks Igor, it works perfectly !
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:23am UTC](https://discuss.elastic.co/t/bulk-indexing-and-count-mismatch/8107/8 "2017-07-06T03:23:22Z")