# Concurrency problem when automatically creating an index **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2015, 5:57pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544 "2015-01-08T17:57:45Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Tom\_6](https://avatars.discourse-cdn.com/v4/letter/t/90db22/32.png) [@Tom\_6](https://discuss.elastic.co/u/Tom_6)\ **Post date:** [January 8, 2015, 5:57pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/1 "2015-01-08T17:57:45Z")
Hi, we'd been using ES for a while now. Specifically version 0.90.3. A couple of months ago we decided to migrate to the latest version which was finally frozen to be 1.4.1. No data migration was necessary because we have a redundant MongoDB, but yesterday we enabled data writing to the new ES cluster. All was running smoothly when we noticed that at o'clock times there were bursts of four or five log messages of the following kinds: Error indexing None into index ind-analytics-2015.01.08. Total elapsed time: 1065 ms. org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (acquire index lock) within 1s at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0\_17] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0\_17] at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: [ind-2015.01.08.00][0] Not enough active copies to meet write consistency of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index {[ind-2015.01.08.00][search][...]} at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776) ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507) ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0\_17] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0\_17] at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] This occurs at o'clock times because we write over hour-based indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and so on. With 0.90.3 version of ES, automatic index creation was working flawlessly (with no complaints) but the new version doesn't seem to handle that feature very well. It looks like, when all those concurrent writes competes to be the first to create the index, all but one fails. Of course we could just create such indices manually to avoid this situation altogether, but this would only be a workaround for a feature that previously worked. Also, we use ES through the native Java client and the configuration for all our indices is settings = { number\_of\_shards = 5, number\_of\_replicas = 2 } Any ideas? Thanks in advance, Tom; -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 8, 2015, 7:19pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/2 "2015-01-08T19:19:43Z")
How many nodes do you have in the cluster? Jörg On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi.despegar@gmail.com](mailto:tarossi.despegar@gmail.com) wrote: > Hi, we'd been using ES for a while now. Specifically version 0.90.3. A > couple of months ago we decided to migrate to the latest version which was > finally frozen to be 1.4.1. No data migration was necessary because we have > a redundant MongoDB, but yesterday we enabled data writing to the new ES > cluster. All was running smoothly when we noticed that at o'clock times > there were bursts of four or five log messages of the following kinds: > > Error indexing None into index ind-analytics-2015.01.08. Total elapsed > time: 1065 ms. > org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: > failed to process cluster event (acquire index lock) within 1s > at > org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0\_17] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0\_17] > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - Scattering > to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > {[ind-2015.01.08.00][search][...]} > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0\_17] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0\_17] > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > This occurs at o'clock times because we write over hour-based indices. For > example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > so on. > > With 0.90.3 version of ES, automatic index creation was working flawlessly > (with no complaints) but the new version doesn't seem to handle that > feature very well. It looks like, when all those concurrent writes competes > to be the first to create the index, all but one fails. Of course we could > just create such indices manually to avoid this situation altogether, but > this would only be a workaround for a feature that previously worked. > > Also, we use ES through the native Java client and the configuration for > all our indices is > > settings = { > number\_of\_shards = 5, > number\_of\_replicas = 2 > } > > Any ideas? > > Thanks in advance, > Tom; > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEL7i\_6Nugjw4XDMAuAK9o6a14%2BDiah9wA37gCtpmf%3DwQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEL7i_6Nugjw4XDMAuAK9o6a14%2BDiah9wA37gCtpmf%3DwQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tom\_6](https://avatars.discourse-cdn.com/v4/letter/t/90db22/32.png) [@Tom\_6](https://discuss.elastic.co/u/Tom_6)\ **Post date:** [January 8, 2015, 7:23pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/3 "2015-01-08T19:23:12Z")
4 El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > How many nodes do you have in the cluster? > > Jörg > > On Thu, Jan 8, 2015 at 6:57 PM, Tom \<[tarossi....@gmail.com](mailto:tarossi....@gmail.com) \\> > wrote: > > > Hi, we'd been using ES for a while now. Specifically version 0.90.3. A > > couple of months ago we decided to migrate to the latest version which was > > finally frozen to be 1.4.1. No data migration was necessary because we have > > a redundant MongoDB, but yesterday we enabled data writing to the new ES > > cluster. All was running smoothly when we noticed that at o'clock times > > there were bursts of four or five log messages of the following kinds: > > > > Error indexing None into index ind-analytics-2015.01.08. Total elapsed > > time: 1065 ms. > > org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: > > failed to process cluster event (acquire index lock) within 1s > > at > > org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > ~[na:1.7.0\_17] > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > ~[na:1.7.0\_17] > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > Scattering to failed in 1043ms > > org.elasticsearch.action.UnavailableShardsException: [ind-2015.01.08.00][0] > > Not enough active copies to meet write consistency of [QUORUM] (have 1, > > needed 2). Timeout: [1s], request: index {[ind-2015.01.08.00][search][...]} > > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776) > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507) > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > ~[na:1.7.0\_17] > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > ~[na:1.7.0\_17] > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > This occurs at o'clock times because we write over hour-based indices. > > For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > so on. > > > > With 0.90.3 version of ES, automatic index creation was working > > flawlessly (with no complaints) but the new version doesn't seem to handle > > that feature very well. It looks like, when all those concurrent writes > > competes to be the first to create the index, all but one fails. Of course > > we could just create such indices manually to avoid this situation > > altogether, but this would only be a workaround for a feature that > > previously worked. > > > > Also, we use ES through the native Java client and the configuration for > > all our indices is > > > > settings = { > > number\_of\_shards = 5, > > number\_of\_replicas = 2 > > } > > > > Any ideas? > > > > Thanks in advance, > > Tom; > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/61a15d1a-02f6-484d-8ce7-862bfe427f17%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/61a15d1a-02f6-484d-8ce7-862bfe427f17%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tom\_6](https://avatars.discourse-cdn.com/v4/letter/t/90db22/32.png) [@Tom\_6](https://discuss.elastic.co/u/Tom_6)\ **Post date:** [January 9, 2015, 2:17pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/4 "2015-01-09T14:17:17Z")
Also, we have another cluster (for different purposes) that has 3 nodes but we didn't experience such errors with it (for this ES we create indices on a daily basis). El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > 4 > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > How many nodes do you have in the cluster? > > > > Jörg > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > Hi, we'd been using ES for a while now. Specifically version 0.90.3. A > > > couple of months ago we decided to migrate to the latest version which was > > > finally frozen to be 1.4.1. No data migration was necessary because we have > > > a redundant MongoDB, but yesterday we enabled data writing to the new ES > > > cluster. All was running smoothly when we noticed that at o'clock times > > > there were bursts of four or five log messages of the following kinds: > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total elapsed > > > time: 1065 ms. > > > org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: > > > failed to process cluster event (acquire index lock) within 1s > > > at > > > org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > ~[na:1.7.0\_17] > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > ~[na:1.7.0\_17] > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > Scattering to failed in 1043ms > > > org.elasticsearch.action.UnavailableShardsException: [ind-2015.01.08.00][0] > > > Not enough active copies to meet write consistency of [QUORUM] (have 1, > > > needed 2). Timeout: [1s], request: index {[ind-2015.01.08.00][search][...]} > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseFailureIfHaveNotEnoughActiveShardCopies(TransportShardReplicationOperationAction.java:776) > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:507) > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > ~[na:1.7.0\_17] > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > ~[na:1.7.0\_17] > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > This occurs at o'clock times because we write over hour-based indices. > > > For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > so on. > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > that feature very well. It looks like, when all those concurrent writes > > > competes to be the first to create the index, all but one fails. Of course > > > we could just create such indices manually to avoid this situation > > > altogether, but this would only be a workaround for a feature that > > > previously worked. > > > > > > Also, we use ES through the native Java client and the configuration for > > > all our indices is > > > > > > settings = { > > > number\_of\_shards = 5, > > > number\_of\_replicas = 2 > > > } > > > > > > Any ideas? > > > > > > Thanks in advance, > > > Tom; > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 9, 2015, 4:25pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/5 "2015-01-09T16:25:53Z")
Exactly, with 3 nodes, the error will be gone. Please, always use an odd number of data nodes, in particular with replica > 0, in order not to confuse ES quorum formula, and also to avoid split > brains with minimun\_master\_nodes Jörg On Fri, Jan 9, 2015 at 3:17 PM, Tom [tarossi.despegar@gmail.com](mailto:tarossi.despegar@gmail.com) wrote: > Also, we have another cluster (for different purposes) that has 3 nodes > but we didn't experience such errors with it (for this ES we create indices > on a daily basis). > > El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > > > 4 > > > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > > > How many nodes do you have in the cluster? > > > > > > Jörg > > > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > Hi, we'd been using ES for a while now. Specifically version 0.90.3. A > > > > couple of months ago we decided to migrate to the latest version which was > > > > finally frozen to be 1.4.1. No data migration was necessary because we have > > > > a redundant MongoDB, but yesterday we enabled data writing to the new ES > > > > cluster. All was running smoothly when we noticed that at o'clock times > > > > there were bursts of four or five log messages of the following kinds: > > > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total elapsed > > > > time: 1065 ms. org.elasticsearch.cluster.metadata. > > > > ProcessClusterEventTimeoutException: failed to process cluster event > > > > (acquire index lock) within 1s > > > > at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.run( > > > > MetaDataCreateIndexService.java:148) ~[org.elasticsearch. > > > > elasticsearch-1.4.1.jar:na] > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > ~[na:1.7.0\_17] > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > ~[na:1.7.0\_17] > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > > Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > > > > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > > > > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > > > > {[ind-2015.01.08.00][search][...]} > > > > at org.elasticsearch.action.support.replication. > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > at org.elasticsearch.action.support.replication. > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > raiseFailureIfHaveNotEnoughActiveShardCopies( > > > > TransportShardReplicationOperationAction.java:776) ~[org.elasticsearch. > > > > elasticsearch-1.4.1.jar:na] > > > > at org.elasticsearch.action.support.replication. > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > performOnPrimary(TransportShardReplicationOperationAction.java:507) > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > at org.elasticsearch.action.support.replication. > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction$1. > > > > run(TransportShardReplicationOperationAction.java:419) > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > ~[na:1.7.0\_17] > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > ~[na:1.7.0\_17] > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > This occurs at o'clock times because we write over hour-based indices. > > > > For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > > so on. > > > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > > that feature very well. It looks like, when all those concurrent writes > > > > competes to be the first to create the index, all but one fails. Of course > > > > we could just create such indices manually to avoid this situation > > > > altogether, but this would only be a workaround for a feature that > > > > previously worked. > > > > > > > > Also, we use ES through the native Java client and the configuration > > > > for all our indices is > > > > > > > > settings = { > > > > number\_of\_shards = 5, > > > > number\_of\_replicas = 2 > > > > } > > > > > > > > Any ideas? > > > > > > > > Thanks in advance, > > > > Tom; > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tomas\_Andres\_Rossi](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@Tomas\_Andres\_Rossi](https://discuss.elastic.co/u/Tomas_Andres_Rossi)\ **Post date:** [January 9, 2015, 6:16pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/6 "2015-01-09T18:16:53Z")
We enlarged our cluster to 5 nodes and now the QUORUM error message seems to have disappeared. "failed to process cluster event (acquire index lock) within 1s" kind of messages are still happening though. ☹ Tom; On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi [trossi@despegar.com](mailto:trossi@despegar.com) wrote: > We enlarged our cluster to 5 nodes and now the QUORUM error message seems > to have disappeared. > "failed to process cluster event (acquire index lock) within 1s" kind of > messages are still happening though. > > ☹ > > Tom; > > On Fri, Jan 9, 2015 at 1:25 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > Exactly, with 3 nodes, the error will be gone. > > > > Please, always use an odd number of data nodes, in particular with > > replica \> 0, in order not to confuse ES quorum formula, and also to avoid > > split brains with minimun\_master\_nodes > > > > Jörg > > > > On Fri, Jan 9, 2015 at 3:17 PM, Tom [tarossi.despegar@gmail.com](mailto:tarossi.despegar@gmail.com) wrote: > > > > > Also, we have another cluster (for different purposes) that has 3 nodes > > > but we didn't experience such errors with it (for this ES we create indices > > > on a daily basis). > > > > > > El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > > > > > > > 4 > > > > > > > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > > > > > > > How many nodes do you have in the cluster? > > > > > > > > > > Jörg > > > > > > > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > Hi, we'd been using ES for a while now. Specifically version 0.90.3. > > > > > > A couple of months ago we decided to migrate to the latest version which > > > > > > was finally frozen to be 1.4.1. No data migration was necessary because we > > > > > > have a redundant MongoDB, but yesterday we enabled data writing to the new > > > > > > ES cluster. All was running smoothly when we noticed that at o'clock times > > > > > > there were bursts of four or five log messages of the following kinds: > > > > > > > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total > > > > > > elapsed time: 1065 ms. org.elasticsearch.cluster.metadata. > > > > > > ProcessClusterEventTimeoutException: failed to process cluster event > > > > > > (acquire index lock) within 1s > > > > > > at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1. > > > > > > run(MetaDataCreateIndexService.java:148) ~[org.elasticsearch. > > > > > > elasticsearch-1.4.1.jar:na] > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > ~[na:1.7.0\_17] > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > ~[na:1.7.0\_17] > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > > > > Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > > > > > > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > > > > > > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > > > > > > {[ind-2015.01.08.00][search][...]} > > > > > > at org.elasticsearch.action.support.replication. > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > at org.elasticsearch.action.support.replication. > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > raiseFailureIfHaveNotEnoughActiveShardCopies( > > > > > > TransportShardReplicationOperationAction.java:776) > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > at org.elasticsearch.action.support.replication. > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > performOnPrimary(TransportShardReplicationOperationAction.java:507) > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > at org.elasticsearch.action.support.replication. > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction$1. > > > > > > run(TransportShardReplicationOperationAction.java:419) > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > ~[na:1.7.0\_17] > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > ~[na:1.7.0\_17] > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > This occurs at o'clock times because we write over hour-based > > > > > > indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > > > > so on. > > > > > > > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > > > > that feature very well. It looks like, when all those concurrent writes > > > > > > competes to be the first to create the index, all but one fails. Of course > > > > > > we could just create such indices manually to avoid this situation > > > > > > altogether, but this would only be a workaround for a feature that > > > > > > previously worked. > > > > > > > > > > > > Also, we use ES through the native Java client and the configuration > > > > > > for all our indices is > > > > > > > > > > > > settings = { > > > > > > number\_of\_shards = 5, > > > > > > number\_of\_replicas = 2 > > > > > > } > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > Thanks in advance, > > > > > > Tom; > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0% > > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Tom; -- Tom; -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 9, 2015, 6:52pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/7 "2015-01-09T18:52:50Z")
It seems there are more than one process trying to create the index, it that possible? Jörg On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi [trossi@despegar.com](mailto:trossi@despegar.com) wrote: > We enlarged our cluster to 5 nodes and now the QUORUM error message seems > to have disappeared. > "failed to process cluster event (acquire index lock) within 1s" kind of > messages are still happening though. > > ☹ > > Tom; > > On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi [trossi@despegar.com](mailto:trossi@despegar.com) > wrote: > > > We enlarged our cluster to 5 nodes and now the QUORUM error message seems > > to have disappeared. > > "failed to process cluster event (acquire index lock) within 1s" kind of > > messages are still happening though. > > > > ☹ > > > > Tom; > > > > On Fri, Jan 9, 2015 at 1:25 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > Exactly, with 3 nodes, the error will be gone. > > > > > > Please, always use an odd number of data nodes, in particular with > > > replica \> 0, in order not to confuse ES quorum formula, and also to avoid > > > split brains with minimun\_master\_nodes > > > > > > Jörg > > > > > > On Fri, Jan 9, 2015 at 3:17 PM, Tom [tarossi.despegar@gmail.com](mailto:tarossi.despegar@gmail.com) wrote: > > > > > > > Also, we have another cluster (for different purposes) that has 3 nodes > > > > but we didn't experience such errors with it (for this ES we create indices > > > > on a daily basis). > > > > > > > > El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > > > > > > > > > 4 > > > > > > > > > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > > > > > > > > > How many nodes do you have in the cluster? > > > > > > > > > > > > Jörg > > > > > > > > > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > > > Hi, we'd been using ES for a while now. Specifically version 0.90.3. > > > > > > > A couple of months ago we decided to migrate to the latest version which > > > > > > > was finally frozen to be 1.4.1. No data migration was necessary because we > > > > > > > have a redundant MongoDB, but yesterday we enabled data writing to the new > > > > > > > ES cluster. All was running smoothly when we noticed that at o'clock times > > > > > > > there were bursts of four or five log messages of the following kinds: > > > > > > > > > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total > > > > > > > elapsed time: 1065 ms. org.elasticsearch.cluster.metadata. > > > > > > > ProcessClusterEventTimeoutException: failed to process cluster > > > > > > > event (acquire index lock) within 1s > > > > > > > at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1. > > > > > > > run(MetaDataCreateIndexService.java:148) ~[org.elasticsearch. > > > > > > > elasticsearch-1.4.1.jar:na] > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > ~[na:1.7.0\_17] > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > ~[na:1.7.0\_17] > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > > > > > Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > > > > > > > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > > > > > > > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > > > > > > > {[ind-2015.01.08.00][search][...]} > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > > retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > > raiseFailureIfHaveNotEnoughActiveShardCopies( > > > > > > > TransportShardReplicationOperationAction.java:776) > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > > performOnPrimary(TransportShardReplicationOperationAction.java:507) > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > TransportShardReplicationOperationAction$ > > > > > > > AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > ~[na:1.7.0\_17] > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > ~[na:1.7.0\_17] > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > This occurs at o'clock times because we write over hour-based > > > > > > > indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > > > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > > > > > so on. > > > > > > > > > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > > > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > > > > > that feature very well. It looks like, when all those concurrent writes > > > > > > > competes to be the first to create the index, all but one fails. Of course > > > > > > > we could just create such indices manually to avoid this situation > > > > > > > altogether, but this would only be a workaround for a feature that > > > > > > > previously worked. > > > > > > > > > > > > > > Also, we use ES through the native Java client and the configuration > > > > > > > for all our indices is > > > > > > > > > > > > > > settings = { > > > > > > > number\_of\_shards = 5, > > > > > > > number\_of\_replicas = 2 > > > > > > > } > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > Thanks in advance, > > > > > > > Tom; > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-](https://groups.google.com/d/msgid/elasticsearch/4deefb09-) > > > > > > > bed1-499a-b9fc-3ed4d78fc4c0%[40googlegroups.com](http://40googlegroups.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Tom; > > -- > Tom; > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG9QZ14j%2Ba96pfP8BJvLqJADRxTyYGgnqM5DAwS%3Dbt0nQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG9QZ14j%2Ba96pfP8BJvLqJADRxTyYGgnqM5DAwS%3Dbt0nQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tomas\_Andres\_Rossi](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@Tomas\_Andres\_Rossi](https://discuss.elastic.co/u/Tomas_Andres_Rossi)\ **Post date:** [January 9, 2015, 7:13pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/8 "2015-01-09T19:13:52Z")
Well yes. We also have a cluster for the app where each node talks to the elastic cluster independently. Remember that we are not creating the index manually. Each app node issues an index operation on an index that may yet not exist and we expect ES to take care of the index creation on demand. Many processes may issue the same indexing operation on the ES cluster "simultaneously" and only one of them must succeed in triggering the index creation. Tom; El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió: > It seems there are more than one process trying to create the index, it > that possible? > > Jörg > > On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi \<[tro...@despegar.com](mailto:tro...@despegar.com) > \\> wrote: > > > We enlarged our cluster to 5 nodes and now the QUORUM error message seems > > to have disappeared. > > "failed to process cluster event (acquire index lock) within 1s" kind of > > messages are still happening though. > > > > ☹ > > > > Tom; > > > > On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi \<[tro...@despegar.com](mailto:tro...@despegar.com) > > \\> wrote: > > > > > We enlarged our cluster to 5 nodes and now the QUORUM error message > > > seems to have disappeared. > > > "failed to process cluster event (acquire index lock) within 1s" kind > > > of messages are still happening though. > > > > > > ☹ > > > > > > Tom; > > > > > > On Fri, Jan 9, 2015 at 1:25 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \ \< > > > [joerg...@gmail.com](mailto:joerg...@gmail.com) \\> wrote: > > > > > > > Exactly, with 3 nodes, the error will be gone. > > > > > > > > Please, always use an odd number of data nodes, in particular with > > > > replica \> 0, in order not to confuse ES quorum formula, and also to avoid > > > > split brains with minimun\_master\_nodes > > > > > > > > Jörg > > > > > > > > On Fri, Jan 9, 2015 at 3:17 PM, Tom \<[tarossi....@gmail.com](mailto:tarossi....@gmail.com) > > > > \\> wrote: > > > > > > > > > Also, we have another cluster (for different purposes) that has 3 > > > > > nodes but we didn't experience such errors with it (for this ES we create > > > > > indices on a daily basis). > > > > > > > > > > El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > > > > > > > > > > > 4 > > > > > > > > > > > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > > > > > > > > > > > How many nodes do you have in the cluster? > > > > > > > > > > > > > > Jörg > > > > > > > > > > > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > > > > > Hi, we'd been using ES for a while now. Specifically version > > > > > > > > 0.90.3. A couple of months ago we decided to migrate to the latest version > > > > > > > > which was finally frozen to be 1.4.1. No data migration was necessary > > > > > > > > because we have a redundant MongoDB, but yesterday we enabled data writing > > > > > > > > to the new ES cluster. All was running smoothly when we noticed that at > > > > > > > > o'clock times there were bursts of four or five log messages of the > > > > > > > > following kinds: > > > > > > > > > > > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total > > > > > > > > elapsed time: 1065 ms. org.elasticsearch.cluster.metadata. > > > > > > > > ProcessClusterEventTimeoutException: failed to process cluster > > > > > > > > event (acquire index lock) within 1s > > > > > > > > at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1. > > > > > > > > run(MetaDataCreateIndexService.java:148) ~[org.elasticsearch. > > > > > > > > elasticsearch-1.4.1.jar:na] > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > > > > > > Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > > > > > > > > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > > > > > > > > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > > > > > > > > {[ind-2015.01.08.00][search][...]} > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > > > retryBecauseUnavailable(TransportShardReplicationOperationAction.java:784) > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > > > raiseFailureIfHaveNotEnoughActiveShardCopies( > > > > > > > > TransportShardReplicationOperationAction.java:776) > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction. > > > > > > > > performOnPrimary(TransportShardReplicationOperationAction.java:507) > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > TransportShardReplicationOperationAction$ > > > > > > > > AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > This occurs at o'clock times because we write over hour-based > > > > > > > > indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > > > > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > > > > > > so on. > > > > > > > > > > > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > > > > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > > > > > > that feature very well. It looks like, when all those concurrent writes > > > > > > > > competes to be the first to create the index, all but one fails. Of course > > > > > > > > we could just create such indices manually to avoid this situation > > > > > > > > altogether, but this would only be a workaround for a feature that > > > > > > > > previously worked. > > > > > > > > > > > > > > > > Also, we use ES through the native Java client and the > > > > > > > > configuration for all our indices is > > > > > > > > > > > > > > > > settings = { > > > > > > > > number\_of\_shards = 5, > > > > > > > > number\_of\_replicas = 2 > > > > > > > > } > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > Tom; > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-](https://groups.google.com/d/msgid/elasticsearch/4deefb09-) > > > > > > > > bed1-499a-b9fc-3ed4d78fc4c0%[40googlegroups.com](http://40googlegroups.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > Tom; > > > > -- > > Tom; > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/86866b0c-329a-4c94-b9b8-9393735f6878%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86866b0c-329a-4c94-b9b8-9393735f6878%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tomas\_Andres\_Rossi](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@Tomas\_Andres\_Rossi](https://discuss.elastic.co/u/Tomas_Andres_Rossi)\ **Post date:** [January 9, 2015, 7:16pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/9 "2015-01-09T19:16:00Z")
Sorry, didn't mean to say "the same indexing operation" but multiple indexing operations (distinct data) on the same non-existent index. El viernes, 9 de enero de 2015, 16:13:52 (UTC-3), Tom escribió: > Well yes. We also have a cluster for the app where each node talks to the > elastic cluster independently. > > Remember that we are not creating the index manually. Each app node issues > an index operation on an index that may yet not exist and we expect ES to > take care of the index creation on demand. Many processes may issue the > same indexing operation on the ES cluster "simultaneously" and only one of > them must succeed in triggering the index creation. > > Tom; > > El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió: > > > It seems there are more than one process trying to create the index, it > > that possible? > > > > Jörg > > > > On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi [tro...@despegar.com](mailto:tro...@despegar.com) > > wrote: > > > > > We enlarged our cluster to 5 nodes and now the QUORUM error message > > > seems to have disappeared. > > > "failed to process cluster event (acquire index lock) within 1s" kind > > > of messages are still happening though. > > > > > > ☹ > > > > > > Tom; > > > > > > On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi [tro...@despegar.com](mailto:tro...@despegar.com) > > > wrote: > > > > > > > We enlarged our cluster to 5 nodes and now the QUORUM error message > > > > seems to have disappeared. > > > > "failed to process cluster event (acquire index lock) within 1s" kind > > > > of messages are still happening though. > > > > > > > > ☹ > > > > > > > > Tom; > > > > > > > > On Fri, Jan 9, 2015 at 1:25 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) [joerg...@gmail.com](mailto:joerg...@gmail.com) > > > > wrote: > > > > > > > > > Exactly, with 3 nodes, the error will be gone. > > > > > > > > > > Please, always use an odd number of data nodes, in particular with > > > > > replica \> 0, in order not to confuse ES quorum formula, and also to avoid > > > > > split brains with minimun\_master\_nodes > > > > > > > > > > Jörg > > > > > > > > > > On Fri, Jan 9, 2015 at 3:17 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > Also, we have another cluster (for different purposes) that has 3 > > > > > > nodes but we didn't experience such errors with it (for this ES we create > > > > > > indices on a daily basis). > > > > > > > > > > > > El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > > > > > > > > > > > > > 4 > > > > > > > > > > > > > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > > > > > > > > > > > > > How many nodes do you have in the cluster? > > > > > > > > > > > > > > > > Jörg > > > > > > > > > > > > > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > > > > > > > Hi, we'd been using ES for a while now. Specifically version > > > > > > > > > 0.90.3. A couple of months ago we decided to migrate to the latest version > > > > > > > > > which was finally frozen to be 1.4.1. No data migration was necessary > > > > > > > > > because we have a redundant MongoDB, but yesterday we enabled data writing > > > > > > > > > to the new ES cluster. All was running smoothly when we noticed that at > > > > > > > > > o'clock times there were bursts of four or five log messages of the > > > > > > > > > following kinds: > > > > > > > > > > > > > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total > > > > > > > > > elapsed time: 1065 ms. org.elasticsearch.cluster.metadata. > > > > > > > > > ProcessClusterEventTimeoutException: failed to process cluster > > > > > > > > > event (acquire index lock) within 1s > > > > > > > > > at org.elasticsearch.cluster.metadata. > > > > > > > > > MetaDataCreateIndexService$1.run(MetaDataCreateIndexService.java:148) > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > > > > > > > Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > > > > > > > > > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > > > > > > > > > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > > > > > > > > > {[ind-2015.01.08.00][search][...]} > > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > > TransportShardReplicationOperationAction$ > > > > > > > > > AsyncShardOperationAction.retryBecauseUnavailable( > > > > > > > > > TransportShardReplicationOperationAction.java:784) > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > > TransportShardReplicationOperationAction$ > > > > > > > > > AsyncShardOperationAction.raiseFailureIfHaveNotEnoughAct > > > > > > > > > iveShardCopies(TransportShardReplicationOperationAction.java:776) > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > > TransportShardReplicationOperationAction$ > > > > > > > > > AsyncShardOperationAction.performOnPrimary( > > > > > > > > > TransportShardReplicationOperationAction.java:507) > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > at org.elasticsearch.action.support.replication. > > > > > > > > > TransportShardReplicationOperationAction$ > > > > > > > > > AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419) > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > > > This occurs at o'clock times because we write over hour-based > > > > > > > > > indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > > > > > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > > > > > > > so on. > > > > > > > > > > > > > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > > > > > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > > > > > > > that feature very well. It looks like, when all those concurrent writes > > > > > > > > > competes to be the first to create the index, all but one fails. Of course > > > > > > > > > we could just create such indices manually to avoid this situation > > > > > > > > > altogether, but this would only be a workaround for a feature that > > > > > > > > > previously worked. > > > > > > > > > > > > > > > > > > Also, we use ES through the native Java client and the > > > > > > > > > configuration for all our indices is > > > > > > > > > > > > > > > > > > settings = { > > > > > > > > > number\_of\_shards = 5, > > > > > > > > > number\_of\_replicas = 2 > > > > > > > > > } > > > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > Tom; > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-](https://groups.google.com/d/msgid/elasticsearch/4deefb09-) > > > > > > > > > bed1-499a-b9fc-3ed4d78fc4c0%[40googlegroups.com](http://40googlegroups.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > Tom; > > > > > > -- > > > Tom; > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 10, 2015, 11:15pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/10 "2015-01-10T23:15:56Z")
I think you can safely ignore "failed to process cluster event (acquire index lock) within 1s" in that case. These messages come from index creation requests that are submitted concurrently - only one request can succeed, the other will get stuck. Jörg On Fri, Jan 9, 2015 at 8:16 PM, Tom [trossi@despegar.com](mailto:trossi@despegar.com) wrote: > Sorry, didn't mean to say "the same indexing operation" but multiple > indexing operations (distinct data) on the same non-existent index. > > El viernes, 9 de enero de 2015, 16:13:52 (UTC-3), Tom escribió: > > > Well yes. We also have a cluster for the app where each node talks to the > > elastic cluster independently. > > > > Remember that we are not creating the index manually. Each app node > > issues an index operation on an index that may yet not exist and we expect > > ES to take care of the index creation on demand. Many processes may issue > > the same indexing operation on the ES cluster "simultaneously" and only one > > of them must succeed in triggering the index creation. > > > > Tom; > > > > El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió: > > > > > It seems there are more than one process trying to create the index, it > > > that possible? > > > > > > Jörg > > > > > > On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi [tro...@despegar.com](mailto:tro...@despegar.com) > > > wrote: > > > > > > > We enlarged our cluster to 5 nodes and now the QUORUM error message > > > > seems to have disappeared. > > > > "failed to process cluster event (acquire index lock) within 1s" kind > > > > of messages are still happening though. > > > > > > > > ☹ > > > > > > > > Tom; > > > > > > > > On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi \<[tro...@despegar.com](mailto:tro...@despegar.com) > > > > > > > > > wrote: > > > > > > > > > We enlarged our cluster to 5 nodes and now the QUORUM error message > > > > > seems to have disappeared. > > > > > "failed to process cluster event (acquire index lock) within 1s" kind > > > > > of messages are still happening though. > > > > > > > > > > ☹ > > > > > > > > > > Tom; > > > > > > > > > > On Fri, Jan 9, 2015 at 1:25 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<[joerg...@gmail.com](mailto:joerg...@gmail.com) > > > > > > > > > > > wrote: > > > > > > > > > > > Exactly, with 3 nodes, the error will be gone. > > > > > > > > > > > > Please, always use an odd number of data nodes, in particular with > > > > > > replica \> 0, in order not to confuse ES quorum formula, and also to avoid > > > > > > split brains with minimun\_master\_nodes > > > > > > > > > > > > Jörg > > > > > > > > > > > > On Fri, Jan 9, 2015 at 3:17 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > > > Also, we have another cluster (for different purposes) that has 3 > > > > > > > nodes but we didn't experience such errors with it (for this ES we create > > > > > > > indices on a daily basis). > > > > > > > > > > > > > > El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > > > > > > > > > > > > > > > 4 > > > > > > > > > > > > > > > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > > > > > > > > > > > > > > > How many nodes do you have in the cluster? > > > > > > > > > > > > > > > > > > Jörg > > > > > > > > > > > > > > > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > > > > > > > > > Hi, we'd been using ES for a while now. Specifically version > > > > > > > > > > 0.90.3. A couple of months ago we decided to migrate to the latest version > > > > > > > > > > which was finally frozen to be 1.4.1. No data migration was necessary > > > > > > > > > > because we have a redundant MongoDB, but yesterday we enabled data writing > > > > > > > > > > to the new ES cluster. All was running smoothly when we noticed that at > > > > > > > > > > o'clock times there were bursts of four or five log messages of the > > > > > > > > > > following kinds: > > > > > > > > > > > > > > > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total > > > > > > > > > > elapsed time: 1065 ms. org.elasticsearch.cluster.metadata. > > > > > > > > > > ProcessClusterEventTimeoutException: failed to process cluster > > > > > > > > > > event (acquire index lock) within 1s > > > > > > > > > > at org.elasticsearch.cluster.metadata.MetaDataCreateIndexServic > > > > > > > > > > e$1.run(MetaDataCreateIndexService.java:148) ~[org.elasticsearch. > > > > > > > > > > elasticsearch-1.4.1.jar:na] > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > > > > > > > > Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > > > > > > > > > > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > > > > > > > > > > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > > > > > > > > > > {[ind-2015.01.08.00][search][...]} > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction.retryBec > > > > > > > > > > auseUnavailable(TransportShardReplicationOperationAction.java:784) > > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction.raiseFai > > > > > > > > > > lureIfHaveNotEnoughActiveShardCopies(TransportShardReplicati > > > > > > > > > > onOperationAction.java:776) ~[org.elasticsearch.elasticsea > > > > > > > > > > rch-1.4.1.jar:na] > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction.performO > > > > > > > > > > nPrimary(TransportShardReplicationOperationAction.java:507) > > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction$1.run(Tr > > > > > > > > > > ansportShardReplicationOperationAction.java:419) > > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > > > > > This occurs at o'clock times because we write over hour-based > > > > > > > > > > indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > > > > > > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > > > > > > > > so on. > > > > > > > > > > > > > > > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > > > > > > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > > > > > > > > that feature very well. It looks like, when all those concurrent writes > > > > > > > > > > competes to be the first to create the index, all but one fails. Of course > > > > > > > > > > we could just create such indices manually to avoid this situation > > > > > > > > > > altogether, but this would only be a workaround for a feature that > > > > > > > > > > previously worked. > > > > > > > > > > > > > > > > > > > > Also, we use ES through the native Java client and the > > > > > > > > > > configuration for all our indices is > > > > > > > > > > > > > > > > > > > > settings = { > > > > > > > > > > number\_of\_shards = 5, > > > > > > > > > > number\_of\_replicas = 2 > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > Tom; > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed) > > > > > > > > > > 1-499a-b9fc-3ed4d78fc4c0%[40googlegroups.com](http://40googlegroups.com) > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-) > > > > > > > > > ab02-49bb-ad79-8e47f249e755%[40googlegroups.com](http://40googlegroups.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > topic/elasticsearch/-H-sNVTSYbQ/unsubscribe. > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD% > > > > > > 2BtgypEWd6LP0CrisyFhh%2BTzjKw%[40mail.gmail.com](http://40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > Tom; > > > > > > > > -- > > > > Tom; > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev% > > > > 3D66Z\_6fxc43zKpRUNg\_g%[40mail.gmail.com](http://40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEh2%2BzGP\_iSRSpa9wXaq0mzaRh74cmf9CkZMakk7KuHrg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEh2%2BzGP_iSRSpa9wXaq0mzaRh74cmf9CkZMakk7KuHrg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tomas\_Andres\_Rossi](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@Tomas\_Andres\_Rossi](https://discuss.elastic.co/u/Tomas_Andres_Rossi)\ **Post date:** [January 12, 2015, 2:16pm UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/11 "2015-01-12T14:16:20Z")
¿So you're saying that those indexing operations associated with the unsuccessful create-index requests will however succeed (i.e. all data will be stored)? On Sat, Jan 10, 2015 at 8:15 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I think you can safely ignore "failed to process cluster event (acquire > index lock) within 1s" in that case. These messages come from index > creation requests that are submitted concurrently - only one request can > succeed, the other will get stuck. > > Jörg > > On Fri, Jan 9, 2015 at 8:16 PM, Tom [trossi@despegar.com](mailto:trossi@despegar.com) wrote: > > > Sorry, didn't mean to say "the same indexing operation" but multiple > > indexing operations (distinct data) on the same non-existent index. > > > > El viernes, 9 de enero de 2015, 16:13:52 (UTC-3), Tom escribió: > > > > > Well yes. We also have a cluster for the app where each node talks to > > > the elastic cluster independently. > > > > > > Remember that we are not creating the index manually. Each app node > > > issues an index operation on an index that may yet not exist and we expect > > > ES to take care of the index creation on demand. Many processes may issue > > > the same indexing operation on the ES cluster "simultaneously" and only one > > > of them must succeed in triggering the index creation. > > > > > > Tom; > > > > > > El viernes, 9 de enero de 2015, 15:53:01 (UTC-3), Jörg Prante escribió: > > > > > > > It seems there are more than one process trying to create the index, it > > > > that possible? > > > > > > > > Jörg > > > > > > > > On Fri, Jan 9, 2015 at 7:16 PM, Tomas Andres Rossi \<[tro...@despegar.com](mailto:tro...@despegar.com) > > > > > > > > > wrote: > > > > > > > > > We enlarged our cluster to 5 nodes and now the QUORUM error message > > > > > seems to have disappeared. > > > > > "failed to process cluster event (acquire index lock) within 1s" kind > > > > > of messages are still happening though. > > > > > > > > > > ☹ > > > > > > > > > > Tom; > > > > > > > > > > On Fri, Jan 9, 2015 at 3:11 PM, Tomas Andres Rossi \< > > > > > [tro...@despegar.com](mailto:tro...@despegar.com)\> wrote: > > > > > > > > > > > We enlarged our cluster to 5 nodes and now the QUORUM error message > > > > > > seems to have disappeared. > > > > > > "failed to process cluster event (acquire index lock) within 1s" > > > > > > kind of messages are still happening though. > > > > > > > > > > > > ☹ > > > > > > > > > > > > Tom; > > > > > > > > > > > > On Fri, Jan 9, 2015 at 1:25 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \< > > > > > > [joerg...@gmail.com](mailto:joerg...@gmail.com)\> wrote: > > > > > > > > > > > > > Exactly, with 3 nodes, the error will be gone. > > > > > > > > > > > > > > Please, always use an odd number of data nodes, in particular with > > > > > > > replica \> 0, in order not to confuse ES quorum formula, and also to avoid > > > > > > > split brains with minimun\_master\_nodes > > > > > > > > > > > > > > Jörg > > > > > > > > > > > > > > On Fri, Jan 9, 2015 at 3:17 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) wrote: > > > > > > > > > > > > > > > Also, we have another cluster (for different purposes) that has 3 > > > > > > > > nodes but we didn't experience such errors with it (for this ES we create > > > > > > > > indices on a daily basis). > > > > > > > > > > > > > > > > El jueves, 8 de enero de 2015, 16:23:12 (UTC-3), Tom escribió: > > > > > > > > > > > > > > > > > 4 > > > > > > > > > > > > > > > > > > El jueves, 8 de enero de 2015 16:19:50 UTC-3, Jörg Prante escribió: > > > > > > > > > > > > > > > > > > > How many nodes do you have in the cluster? > > > > > > > > > > > > > > > > > > > > Jörg > > > > > > > > > > > > > > > > > > > > On Thu, Jan 8, 2015 at 6:57 PM, Tom [tarossi....@gmail.com](mailto:tarossi....@gmail.com) > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, we'd been using ES for a while now. Specifically version > > > > > > > > > > > 0.90.3. A couple of months ago we decided to migrate to the latest version > > > > > > > > > > > which was finally frozen to be 1.4.1. No data migration was necessary > > > > > > > > > > > because we have a redundant MongoDB, but yesterday we enabled data writing > > > > > > > > > > > to the new ES cluster. All was running smoothly when we noticed that at > > > > > > > > > > > o'clock times there were bursts of four or five log messages of the > > > > > > > > > > > following kinds: > > > > > > > > > > > > > > > > > > > > > > Error indexing None into index ind-analytics-2015.01.08. Total > > > > > > > > > > > elapsed time: 1065 ms. org.elasticsearch.cluster.metadata. > > > > > > > > > > > ProcessClusterEventTimeoutException: failed to process cluster > > > > > > > > > > > event (acquire index lock) within 1s > > > > > > > > > > > at org.elasticsearch.cluster.metadata.MetaDataCreateIndexServic > > > > > > > > > > > e$1.run(MetaDataCreateIndexService.java:148) > > > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > > > > > > > [ForkJoinPool-2-worker-15] c.d.i.p.ActorScatterGatherStrategy - > > > > > > > > > > > Scattering to failed in 1043ms org.elasticsearch.action.UnavailableShardsException: > > > > > > > > > > > [ind-2015.01.08.00][0] Not enough active copies to meet write consistency > > > > > > > > > > > of [QUORUM] (have 1, needed 2). Timeout: [1s], request: index > > > > > > > > > > > {[ind-2015.01.08.00][search][...]} > > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction.retryBec > > > > > > > > > > > auseUnavailable(TransportShardReplicationOperationAction.java:784) > > > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction.raiseFai > > > > > > > > > > > lureIfHaveNotEnoughActiveShardCopies(TransportShardReplicati > > > > > > > > > > > onOperationAction.java:776) ~[org.elasticsearch.elasticsea > > > > > > > > > > > rch-1.4.1.jar:na] > > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction.performO > > > > > > > > > > > nPrimary(TransportShardReplicationOperationAction.java:507) > > > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > > > at org.elasticsearch.action.support.replication.TransportShardR > > > > > > > > > > > eplicationOperationAction$AsyncShardOperationAction$1.run(Tr > > > > > > > > > > > ansportShardReplicationOperationAction.java:419) > > > > > > > > > > > ~[org.elasticsearch.elasticsearch-1.4.1.jar:na] > > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > > > > > > > ~[na:1.7.0\_17] > > > > > > > > > > > at java.lang.Thread.run(Thread.java:722) ~[na:1.7.0\_17] > > > > > > > > > > > > > > > > > > > > > > This occurs at o'clock times because we write over hour-based > > > > > > > > > > > indices. For example, all writes from 18:00:00 to 18:59:59 of 01/08 goes to > > > > > > > > > > > ind-2015.01.08.18. At 19:00:00 all writes will go to ind-2015.01.08.19, and > > > > > > > > > > > so on. > > > > > > > > > > > > > > > > > > > > > > With 0.90.3 version of ES, automatic index creation was working > > > > > > > > > > > flawlessly (with no complaints) but the new version doesn't seem to handle > > > > > > > > > > > that feature very well. It looks like, when all those concurrent writes > > > > > > > > > > > competes to be the first to create the index, all but one fails. Of course > > > > > > > > > > > we could just create such indices manually to avoid this situation > > > > > > > > > > > altogether, but this would only be a workaround for a feature that > > > > > > > > > > > previously worked. > > > > > > > > > > > > > > > > > > > > > > Also, we use ES through the native Java client and the > > > > > > > > > > > configuration for all our indices is > > > > > > > > > > > > > > > > > > > > > > settings = { > > > > > > > > > > > number\_of\_shards = 5, > > > > > > > > > > > number\_of\_replicas = 2 > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > > Tom; > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed) > > > > > > > > > > > 1-499a-b9fc-3ed4d78fc4c0%[40googlegroups.com](http://40googlegroups.com) > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4deefb09-bed1-499a-b9fc-3ed4d78fc4c0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-) > > > > > > > > > > ab02-49bb-ad79-8e47f249e755%[40googlegroups.com](http://40googlegroups.com) > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b052ab5-ab02-49bb-ad79-8e47f249e755%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > topic/elasticsearch/-H-sNVTSYbQ/unsubscribe. > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/](https://groups.google.com/d/msgid/elasticsearch/) > > > > > > > CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw% > > > > > > > [40mail.gmail.com](http://40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFaU96UN8YaguRs%2BMqD%2BtgypEWd6LP0CrisyFhh%2BTzjKw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > Tom; > > > > > > > > > > -- > > > > > Tom; > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev% > > > > > 3D66Z\_6fxc43zKpRUNg\_g%[40mail.gmail.com](http://40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_cjcJNOt-aRZQc\_3u7Xj0Knev%3D66Z\_6fxc43zKpRUNg\_g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_cjcJNOt-aRZQc_3u7Xj0Knev%3D66Z_6fxc43zKpRUNg_g%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6f30f8b-d258-4356-b7ef-ee6a0f9edf3c%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/-H-sNVTSYbQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEh2%2BzGP\_iSRSpa9wXaq0mzaRh74cmf9CkZMakk7KuHrg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEh2%2BzGP_iSRSpa9wXaq0mzaRh74cmf9CkZMakk7KuHrg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEh2%2BzGP\_iSRSpa9wXaq0mzaRh74cmf9CkZMakk7KuHrg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEh2%2BzGP_iSRSpa9wXaq0mzaRh74cmf9CkZMakk7KuHrg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Tom; -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOs2X\_fQjEbOFwkB237vDE1qErKcs1zmzUrWp93L5\_5kLw83zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOs2X_fQjEbOFwkB237vDE1qErKcs1zmzUrWp93L5_5kLw83zA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/concurrency-problem-when-automatically-creating-an-index/21544/12 "2017-07-06T00:39:33Z")