ElasticSearch cannot perform large searches when the index is being updated

My server has a MongoDB database, ElasticSearch, Monstache (keeps the MongoDB database and ElasticSearch index in sync), and a Node.js REST API running at the same time. Users are conducting searches via a web app. The web app sends the search query to the API, which sends request(s) to ElasticSearch to get the search results. Large searches use ElasticSearch's search_after feature to retrieve the results in batches (pagination).

When Monstache is updating the ElasticSearch index, ElasticSearch fails to return any more batches of search results past a certain point. So for example, the API will retrieve 80-90% of the results in batches; then when it tries to retrieve the next batch, only part of the batch is returned; then when it tries to retrieve the next batch, it receives an empty batch and the API crashes.

Is ElasticSearch not designed to search and update an index at the same time? Why can't I successfully search with ElasticSearch while the index is being updated?

search_after doesn't guarantee that it will always work on a consistent snapshot of the data, so if you change the data while it's running then you may retrieve fewer results than you expected. But "crash" is a strong word. What do you mean by "the API crashes"? Can you share the error you're seeing?