Process Cluster Event Timeout Exception


(greenmang0) #1

Hi List,
I am getting this ProcessClusterEventTimeoutException when I try to create
a new index. What could cause it? I have a medium size cluster (EC2 - XL)
with 4 nodes. The cluster status is yellow because of some unassigned nodes
but things are moving along. Any idea why I am getting that error? Is there
any fix/workaround?. Also I have experienced ES is taking so much time to
assign shards, is that IO expensive process?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #2

Hey,

do you have any logfiles you could share in a gist maybe? Can you also post
the elasticsearch and jvm versions you are using and the steps you are
executing to reproduce this problem? Does this happen with an empty cluster
(no data in it) and can be reliable reproduced as well?

thanks!

--Alex

On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar <
abhishek.amberkar@gmail.com> wrote:

Hi List,
I am getting this ProcessClusterEventTimeoutException when I try to create
a new index. What could cause it? I have a medium size cluster (EC2 - XL)
with 4 nodes. The cluster status is yellow because of some unassigned nodes
but things are moving along. Any idea why I am getting that error? Is there
any fix/workaround?. Also I have experienced ES is taking so much time to
assign shards, is that IO expensive process?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(greenmang0) #3

Here's the exception I got - http://pastebin.com/fnmZPt0k

Elasticsearch Version - 0.90.3
JVM Version - Java version "1.7.0_40" Java(TM) SE Runtime Environment
(build 1.7.0_40-b43) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56,
mixed mode)

I am using Oracle Java

JVM flags being used for Elasticsearch -
-Xms11g -Xmx11g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-XX:+CMSPermGenSweepingEnabled -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled

We have around 1.5k small indexes. I could reproduce the problem, when I
was creating index using Bulk API. (Refresh Interval was set to -1). And
there were thousands of unassigned shards in the cluster. I never tried
this on an empty cluster.

On Sunday, September 15, 2013 6:31:12 PM UTC+5:30, Alexander Reelsen wrote:

Hey,

do you have any logfiles you could share in a gist maybe? Can you also
post the elasticsearch and jvm versions you are using and the steps you are
executing to reproduce this problem? Does this happen with an empty cluster
(no data in it) and can be reliable reproduced as well?

thanks!

--Alex

On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar <abhishek...@gmail.com<javascript:>

wrote:

Hi List,
I am getting this ProcessClusterEventTimeoutException when I try to
create a new index. What could cause it? I have a medium size cluster (EC2

  • XL) with 4 nodes. The cluster status is yellow because of some unassigned
    nodes but things are moving along. Any idea why I am getting that error? Is
    there any fix/workaround?. Also I have experienced ES is taking so much
    time to assign shards, is that IO expensive process?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #4

Hey,

these exceptions look like from some clojure application? I suspect this is
your client. Is there anything in the logfiles of your elasticsearch
instance/cluster?
Also do you have unbounded max open files for the elasticsearch user
configured (you should see this as well in your elasticsearch
instance/cluster log files)?

--Alex

On Mon, Sep 16, 2013 at 10:31 AM, Abhishek Amberkar <
abhishek.amberkar@gmail.com> wrote:

Here's the exception I got - http://pastebin.com/fnmZPt0k

Elasticsearch Version - 0.90.3
JVM Version - Java version "1.7.0_40" Java(TM) SE Runtime Environment
(build 1.7.0_40-b43) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56,
mixed mode)

I am using Oracle Java

JVM flags being used for Elasticsearch -
-Xms11g -Xmx11g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-XX:+CMSPermGenSweepingEnabled -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled

We have around 1.5k small indexes. I could reproduce the problem, when I
was creating index using Bulk API. (Refresh Interval was set to -1). And
there were thousands of unassigned shards in the cluster. I never tried
this on an empty cluster.

On Sunday, September 15, 2013 6:31:12 PM UTC+5:30, Alexander Reelsen wrote:

Hey,

do you have any logfiles you could share in a gist maybe? Can you also
post the elasticsearch and jvm versions you are using and the steps you are
executing to reproduce this problem? Does this happen with an empty cluster
(no data in it) and can be reliable reproduced as well?

thanks!

--Alex

On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar <abhishek...@gmail.com

wrote:

Hi List,
I am getting this ProcessClusterEventTimeoutExce**ption when I try to
create a new index. What could cause it? I have a medium size cluster (EC2

  • XL) with 4 nodes. The cluster status is yellow because of some unassigned
    nodes but things are moving along. Any idea why I am getting that error? Is
    there any fix/workaround?. Also I have experienced ES is taking so much
    time to assign shards, is that IO expensive process?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(greenmang0) #5

Reopening the discussion after two months and continuing.

I didn't have much useful information in log files. I have set max
open files to 999999.

On Mon, Sep 16, 2013 at 7:21 PM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

these exceptions look like from some clojure application? I suspect this is
your client. Is there anything in the logfiles of your elasticsearch
instance/cluster?
Also do you have unbounded max open files for the elasticsearch user
configured (you should see this as well in your elasticsearch
instance/cluster log files)?

--Alex

On Mon, Sep 16, 2013 at 10:31 AM, Abhishek Amberkar
abhishek.amberkar@gmail.com wrote:

Here's the exception I got - http://pastebin.com/fnmZPt0k

Elasticsearch Version - 0.90.3
JVM Version - Java version "1.7.0_40" Java(TM) SE Runtime Environment
(build 1.7.0_40-b43) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56,
mixed mode)

I am using Oracle Java

JVM flags being used for Elasticsearch -
-Xms11g -Xmx11g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-XX:+CMSPermGenSweepingEnabled -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled

We have around 1.5k small indexes. I could reproduce the problem, when I
was creating index using Bulk API. (Refresh Interval was set to -1). And
there were thousands of unassigned shards in the cluster. I never tried this
on an empty cluster.

On Sunday, September 15, 2013 6:31:12 PM UTC+5:30, Alexander Reelsen
wrote:

Hey,

do you have any logfiles you could share in a gist maybe? Can you also
post the elasticsearch and jvm versions you are using and the steps you are
executing to reproduce this problem? Does this happen with an empty cluster
(no data in it) and can be reliable reproduced as well?

thanks!

--Alex

On Sat, Sep 14, 2013 at 2:24 PM, Abhishek Amberkar
abhishek...@gmail.com wrote:

Hi List,
I am getting this ProcessClusterEventTimeoutException when I try to
create a new index. What could cause it? I have a medium size cluster (EC2 -
XL) with 4 nodes. The cluster status is yellow because of some unassigned
nodes but things are moving along. Any idea why I am getting that error? Is
there any fix/workaround?. Also I have experienced ES is taking so much time
to assign shards, is that IO expensive process?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/A05Z3RCWr6o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
With Regards
Abhishek Amberkar

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #6