Thank you @dadoonet .
- We are using ES rest client v5.5.0. But the cluster is at v5.6.2 - This is a pre-production test cluster.
- When do indexing, immediately after that we invoke the refresh API as well. I know it is not the most efficient way to make data available sooner for searching but since our write rates are low, we thought we could live with this inefficiency. If there is a better way to handle this scenario then please let us know.
Based on what you have suggested and my usecase, I cannot use BulkProcesser API. In addition, if, by default, fsync is happening on each write, do I need to invoke refresh API to make the document searchable?
Wanted to add that the behavior I describe above in #2 is actually running live in our production cluster. We were evaluating v5.6.2 so that we can upgrade production to that newer version. For we have a bunch of integration tests that run sequentially. And during that test run, we ran into this issue. We are going to try and create new cluster with v5.5.0 to see if this issue persists or not. We had not seen this issue previously.