Hi,
I have written a Spring Boot based integration test where I use the bulk api to index some documents. This is the main loop of the reindexAll() method:
for (page in pageExports) {
bulkRequest.operations { ops -> ops.index { it
.index(Index.PAGE.indexName)
.id(page.pageId.toString())
.document(page)
} }
}
val result = esClient.bulk(bulkRequest.build())
The result comes back just fine and the took field has the value 16. My understanding was that this call is blocking. (E.g. the request only returns when all the documents have been ingested).
However right after I do the result when I do a count query on the index like so:
val result = esClient.count { it.index(listOf(DwIndex.PAGE.indexName))}.count()
I get 0 as a count, same if I fire a search query. If I wait on breakpoint after the indexing and before the query, or if I put a Thread.sleep(100) in front of the count/search, it is correct.
So my question is what is happening here? Possible explanations that came to mind:
- Bulk is indeed asynchronous, which doesn't make a lot of sense since it has the errors for documents that could not be indexex
- Bulk is synchronous, but after the indexing, it takes some (small amount of) time for the newly indexed documents to be pushed to the shards.
- I'm doing something wrong
What would be a good way to make reindexAll() block until everything is ready, so the contract should be if you call reindexAll() you can instantly start searching. I would not like to add a Thread.sleep() or poll with a search request at the end of the method.
Cheers and thanks!