My server has a MongoDB database, ElasticSearch, Monstache (keeps the MongoDB database and ElasticSearch index in sync), and a Node.js REST API running at the same time. Users are conducting searches via a web app. The web app sends the search query to the API, which sends request(s) to ElasticSearch to get the search results. Large searches use ElasticSearch's search_after feature to retrieve the results in batches (pagination).
When Monstache is updating the ElasticSearch index, ElasticSearch fails to return any more batches of search results past a certain point. So for example, the API will retrieve 80-90% of the results in batches; then when it tries to retrieve the next batch, only part of the batch is returned; then when it tries to retrieve the next batch, it receives an empty batch and the API crashes.
Is ElasticSearch not designed to search and update an index at the same time? Why can't I successfully search with ElasticSearch while the index is being updated?