Primary shard is not active timeout

[LazPix]

I'm using ES 2.4.4 on a linux machine. I have a node js process that reads from a queue and writes in parallel 16 different batches of about 1MB each to Elasticsearch. After each write the process rests for 30ms. Most of the time the data flows to db reasonably well (though it sometimes takes more than 5 seconds per batch to be indexed), but every once in a while, I get an error that says primary shard is not active - status 503. In most of the times, the problem is somehow spontaneously resolved, and the process resumes writing. However, sometimes it stays that way for a very long time, and this is disturbing. After I delete the index, the data starts flowing very fast (about 50ms per batch). However, in a production environment I can't simply delete an index. Is there any tuning I can do to avoid this issue? and is it true that the smaller the index the faster it is to write to it?

Thanks.

[Christian_Dahlqvist]

If you are allowing Elasticsearch to define the document ID the size of the index does in my experience not affect indexing throughput to any great extent. If you however index with externally generated document IDs, each indexing operation is a potential update and depending on how you create your ID, looking up whether the document already exists or not can get slower the larger the index grows.