# Process Cluster Event Timeout Exception **URL:** \ **Category:** Elasticsearch\ **Created:** [September 14, 2013, 12:24pm UTC](https://discuss.elastic.co/t/process-cluster-event-timeout-exception/13606 "2013-09-14T12:24:10Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![greenmang0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greenmang0/32/2000_2.png) [@greenmang0](https://discuss.elastic.co/u/greenmang0)\ **Post date:** [September 14, 2013, 12:24pm UTC](https://discuss.elastic.co/t/process-cluster-event-timeout-exception/13606/1 "2013-09-14T12:24:10Z")
Hi List, I am getting this ProcessClusterEventTimeoutException when I try to create a new index. What could cause it? I have a medium size cluster (EC2 - XL) with 4 nodes. The cluster status is yellow because of some unassigned nodes but things are moving along. Any idea why I am getting that error? Is there any fix/workaround?. Also I have experienced ES is taking so much time to assign shards, is that IO expensive process? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 15, 2013, 1:01pm UTC](https://discuss.elastic.co/t/process-cluster-event-timeout-exception/13606/2 "2013-09-15T13:01:12Z")
Hey, do you have any logfiles you could share in a gist maybe? Can you also post the elasticsearch and jvm versions you are using and the steps you are executing to reproduce this problem? Does this happen with an empty cluster (no data in it) and can be reliable reproduced as well? thanks! --Alex On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar \< [abhishek.amberkar@gmail.com](mailto:abhishek.amberkar@gmail.com)\> wrote: > Hi List, > I am getting this ProcessClusterEventTimeoutException when I try to create > a new index. What could cause it? I have a medium size cluster (EC2 - XL) > with 4 nodes. The cluster status is yellow because of some unassigned nodes > but things are moving along. Any idea why I am getting that error? Is there > any fix/workaround?. Also I have experienced ES is taking so much time to > assign shards, is that IO expensive process? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![greenmang0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greenmang0/32/2000_2.png) [@greenmang0](https://discuss.elastic.co/u/greenmang0)\ **Post date:** [September 16, 2013, 8:31am UTC](https://discuss.elastic.co/t/process-cluster-event-timeout-exception/13606/3 "2013-09-16T08:31:00Z")
Here's the exception I got - [Elasticsearch - Process Cluster Timeout Exception - Pastebin.com](http://pastebin.com/fnmZPt0k) Elasticsearch Version - 0.90.3 JVM Version - Java version "1.7.0\_40" Java(TM) SE Runtime Environment (build 1.7.0\_40-b43) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode) I am using Oracle Java JVM flags being used for Elasticsearch - -Xms11g -Xmx11g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSPermGenSweepingEnabled -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled We have around 1.5k small indexes. I could reproduce the problem, when I was creating index using Bulk API. (Refresh Interval was set to -1). And there were thousands of unassigned shards in the cluster. I never tried this on an empty cluster. On Sunday, September 15, 2013 6:31:12 PM UTC+5:30, Alexander Reelsen wrote: > Hey, > > do you have any logfiles you could share in a gist maybe? Can you also > post the elasticsearch and jvm versions you are using and the steps you are > executing to reproduce this problem? Does this happen with an empty cluster > (no data in it) and can be reliable reproduced as well? > > thanks! > > --Alex > > On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar \<[abhishek...@gmail.com](mailto:abhishek...@gmail.com)\ > > > wrote: > > > Hi List, > > I am getting this ProcessClusterEventTimeoutException when I try to > > create a new index. What could cause it? I have a medium size cluster (EC2 > > > > - XL) with 4 nodes. The cluster status is yellow because of some unassigned > > nodes but things are moving along. Any idea why I am getting that error? Is > > there any fix/workaround?. Also I have experienced ES is taking so much > > time to assign shards, is that IO expensive process? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 16, 2013, 1:51pm UTC](https://discuss.elastic.co/t/process-cluster-event-timeout-exception/13606/4 "2013-09-16T13:51:30Z")
Hey, these exceptions look like from some clojure application? I suspect this is your client. Is there anything in the logfiles of your elasticsearch instance/cluster? Also do you have unbounded max open files for the elasticsearch user configured (you should see this as well in your elasticsearch instance/cluster log files)? --Alex On Mon, Sep 16, 2013 at 10:31 AM, Abhishek Amberkar \< [abhishek.amberkar@gmail.com](mailto:abhishek.amberkar@gmail.com)\> wrote: > Here's the exception I got - [Elasticsearch - Process Cluster Timeout Exception - Pastebin.com](http://pastebin.com/fnmZPt0k) > > Elasticsearch Version - 0.90.3 > JVM Version - Java version "1.7.0\_40" Java(TM) SE Runtime Environment > (build 1.7.0\_40-b43) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, > mixed mode) > > I am using Oracle Java > > JVM flags being used for Elasticsearch - > -Xms11g -Xmx11g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError > -XX:+CMSPermGenSweepingEnabled -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled > > We have around 1.5k small indexes. I could reproduce the problem, when I > was creating index using Bulk API. (Refresh Interval was set to -1). And > there were thousands of unassigned shards in the cluster. I never tried > this on an empty cluster. > > On Sunday, September 15, 2013 6:31:12 PM UTC+5:30, Alexander Reelsen wrote: > > > Hey, > > > > do you have any logfiles you could share in a gist maybe? Can you also > > post the elasticsearch and jvm versions you are using and the steps you are > > executing to reproduce this problem? Does this happen with an empty cluster > > (no data in it) and can be reliable reproduced as well? > > > > thanks! > > > > --Alex > > > > On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar \<[abhishek...@gmail.com](mailto:abhishek...@gmail.com) > > > > > wrote: > > > > > Hi List, > > > I am getting this ProcessClusterEventTimeoutExce\*\*ption when I try to > > > create a new index. What could cause it? I have a medium size cluster (EC2 > > > > > > - XL) with 4 nodes. The cluster status is yellow because of some unassigned > > > nodes but things are moving along. Any idea why I am getting that error? Is > > > there any fix/workaround?. Also I have experienced ES is taking so much > > > time to assign shards, is that IO expensive process? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![greenmang0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greenmang0/32/2000_2.png) [@greenmang0](https://discuss.elastic.co/u/greenmang0)\ **Post date:** [November 10, 2013, 12:30pm UTC](https://discuss.elastic.co/t/process-cluster-event-timeout-exception/13606/5 "2013-11-10T12:30:35Z")
Reopening the discussion after two months and continuing. I didn't have much useful information in log files. I have set max open files to 999999. On Mon, Sep 16, 2013 at 7:21 PM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > Hey, > > these exceptions look like from some clojure application? I suspect this is > your client. Is there anything in the logfiles of your elasticsearch > instance/cluster? > Also do you have unbounded max open files for the elasticsearch user > configured (you should see this as well in your elasticsearch > instance/cluster log files)? > > --Alex > > On Mon, Sep 16, 2013 at 10:31 AM, Abhishek Amberkar > [abhishek.amberkar@gmail.com](mailto:abhishek.amberkar@gmail.com) wrote: > > > Here's the exception I got - [Elasticsearch - Process Cluster Timeout Exception - Pastebin.com](http://pastebin.com/fnmZPt0k) > > > > Elasticsearch Version - 0.90.3 > > JVM Version - Java version "1.7.0\_40" Java(TM) SE Runtime Environment > > (build 1.7.0\_40-b43) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, > > mixed mode) > > > > I am using Oracle Java > > > > JVM flags being used for Elasticsearch - > > -Xms11g -Xmx11g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC > > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 > > -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError > > -XX:+CMSPermGenSweepingEnabled -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled > > > > We have around 1.5k small indexes. I could reproduce the problem, when I > > was creating index using Bulk API. (Refresh Interval was set to -1). And > > there were thousands of unassigned shards in the cluster. I never tried this > > on an empty cluster. > > > > On Sunday, September 15, 2013 6:31:12 PM UTC+5:30, Alexander Reelsen > > wrote: > > > > > Hey, > > > > > > do you have any logfiles you could share in a gist maybe? Can you also > > > post the elasticsearch and jvm versions you are using and the steps you are > > > executing to reproduce this problem? Does this happen with an empty cluster > > > (no data in it) and can be reliable reproduced as well? > > > > > > thanks! > > > > > > --Alex > > > > > > On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar > > > [abhishek...@gmail.com](mailto:abhishek...@gmail.com) wrote: > > > > > > > Hi List, > > > > I am getting this ProcessClusterEventTimeoutException when I try to > > > > create a new index. What could cause it? I have a medium size cluster (EC2 - > > > > XL) with 4 nodes. The cluster status is yellow because of some unassigned > > > > nodes but things are moving along. Any idea why I am getting that error? Is > > > > there any fix/workaround?. Also I have experienced ES is taking so much time > > > > to assign shards, is that IO expensive process? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/A05Z3RCWr6o/unsubscribe](https://groups.google.com/d/topic/elasticsearch/A05Z3RCWr6o/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- With Regards Abhishek Amberkar -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:07am UTC](https://discuss.elastic.co/t/process-cluster-event-timeout-exception/13606/6 "2017-07-06T02:07:56Z")