I'm using ES 2.4.4 on a linux machine. I have a node js process that reads from a queue and writes in parallel 16 different batches of about 1MB each to Elasticsearch. After each write the process rests for 30ms. Most of the time the data flows to db reasonably well (though it sometimes takes more than 5 seconds per batch to be indexed), but every once in a while, I get an error that says primary shard is not active - status 503. In most of the times, the problem is somehow spontaneously resolved, and the process resumes writing. However, sometimes it stays that way for a very long time, and this is disturbing. After I delete the index, the data starts flowing very fast (about 50ms per batch). However, in a production environment I can't simply delete an index. Is there any tuning I can do to avoid this issue? and is it true that the smaller the index the faster it is to write to it?
Thanks.