Detecting the end of a bulk indexing

Elastic Stack Elasticsearch
1

How do I detect the end of a bulk indexing process? In other words, how do I know I am good to go knowing all data are available for search?

I've seen a few old messages on this topic but they seemed related to very specific use cases.

Thank you,
Paolo

2

Short answer: just stick ?refresh on the bulk request.

Long answer: It depends on your use case a bit. One way is to add ?refresh on the bulk request if there is only a single bulk request. If you have a bunch of bulk requests then you can issue a refresh request after all of them. Both of these immediately make the data available - it'll be ready for search as soon as the response comes back.

The downside of explicit refreshes is that they create segments "right now". That means that they can create small segments. If you aren't adding other data to the index and you know you are done they is no downside to forcing a refresh - waiting for one wouldn't make anything better.

If you are adding data to the index at the same time as something you need to be visible then master has ?refresh=wait_for that will let you, "wait for" the action to be made visible by some other refresh. Again, you don't need it if you are the only writer. It is for when you are writing and there will be other writes between yours and the scheduled refresh.

3

Thank you Nik, that is helpful.

And I assume when, for ?refresh=wait_for you say "master" you mean it will be available in v.5 which would be consistent with this issue https://github.com/elastic/elasticsearch/issues/1063

Correct?
Thank you again,
Paolo

4

It didn't make 5.0.0-alpha-3 but it'll go into whatever release of 5.0.0 comes next.