I'm running two elasticsearch nodes (0.18.6), with unicast discovery,
5 shards and 1 replica per index, 747 indices, and 2 types per index.
I restarted elasticsearch on each server (waited about a minute
between them), and checked the cluster state. It was red for 30-40
minutes, then yellow for about an hour, with the "unassigned_shards"
count slowly decreasing. At first, it was recovering 120 shards per
minute, and then it slowed to 12 shards per minute. With approximately
7400 shards, it would take hours to fully recover.
I tried running my application server (Play Framework) during the
recovery process, and found that search results would usually come
back ok, but whenever I tried to index new data, elasticsearch
wouldn't respond. I had the same results with curl -- searching
worked, but elasticsearch didn't respond when trying to add data (or
create an index).
Is it normal for elasticsearch to take so long to recover? How can I
make it faster? Is it normal for ES to not respond to PUT requests
while it's recovering?
My elasticsearch configuration file: