Elasticsearch Bulk upload issue

Elastic Stack Elasticsearch
1

We are facing one issue for the bulk load API we are using for
ES version 2.1.1. This seems to be issue for the case when automatic index
creation takes place in ES. It look like, when all those concurrent writes in
bulk load competes to be the first to create the index, all but one fails. We
want indices to be automatically created day wise based on template defined in
ES. Once the index is created we don’t observe this issue for the existing
indices or even with more data count in bulk load than the one failed with this
error.

Following is the error observed in this case -

RemoteTransportException[[host][host:ip][indices:admin/create]];
nested: ProcessClusterEventTimeoutException[failed to process cluster event
(acquire index lock) within 4s];,index_2016.03.14

Details:

We are using 3 node cluster for ES where number_of_replicas: 2
and number_of_shards: 1 (3 master and 3 data nodes)

We have kept 4 sec as the timeout value for the bulk load which
we can change (increase) if required.

Is the cause of error is automatic index creation or
something else and if this is the existing issue, is there any workaround ?

2

The alternative is to create an index explicitly, with all settings and mappings.

3

Actually we are creating index dynamically, we would like to go with dynamic creation only day wise. Otherwise we will have to write separate program to create those indices everyday!