Exception cause unwrapping ran for 10 levels


(Jason Wee) #1

Hi,

we are using elasticsearch version 0.90.7 and following are gist file.
https://gist.github.com/jasonwee/8282477 not sure what is happening
"Exception cause unwrapping ran for 10 levels" and if it is a concern? The
exception shown after some time. Any idea?

Jason

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9d767e9d-33b9-4c36-b725-5aa3bf6d7a09%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #2

ES is unable to transport an exception back to the client, exceptions are
thrown, passed back and forth, and the procedure is retried until the Java
stack is full (StackOverflowError). Yes, it is a concern, it is a bug, a
stack overflow must not happen.

Is the gist the whole stack trace? I bet you only posted the first few
lines...

In the meantime, can you check if you run the same JVM version at server
and client side? A transport exception can not be deserialized, and this is
typical for a JVM mismatch.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH37nkrvCOdfqpYP_sDD-ChmpiipWLb%3DgMe15LfttUv4w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jason Wee) #3

Is the gist the whole stack trace? I bet you only posted the first few
lines...
log filesize is 427MB and total lines 5463550 and if i paste the entire
file into gist, it would break github limit? After gzip the file, it is
5.9MB. If you have a server, I can upload the gzip log to your server.

server and client both running the same version the entire times.

$ java -version
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)

On Mon, Jan 6, 2014 at 9:33 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

ES is unable to transport an exception back to the client, exceptions are
thrown, passed back and forth, and the procedure is retried until the Java
stack is full (StackOverflowError). Yes, it is a concern, it is a bug, a
stack overflow must not happen.

Is the gist the whole stack trace? I bet you only posted the first few
lines...

In the meantime, can you check if you run the same JVM version at server
and client side? A transport exception can not be deserialized, and this is
typical for a JVM mismatch.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH37nkrvCOdfqpYP_sDD-ChmpiipWLb%3DgMe15LfttUv4w%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itxSN34myLoEu5ofj3HAi0G18bQpeUECT-rXJ7wq-k5%3DEQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #4

I was not speaking of the whole logfile, but the whole stacktrace, just to
make sure where the culprit started. If there was no OOM or other things
around, it would be surprising.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFi-T8iiKTULr_S-7OddJnAoGGF4%3Db6reB3vGejDArgnQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jason Wee) #5

Hi Jörg,

Sure, my bad. There is no OOM and here is the second gist,
https://gist.github.com/jasonwee/8294514 . The exceptions are repetitive,
so that explains large number of lines in the log file.

Does the second gist help to determine the problem? I can get the log if it
does not tell where might be the cause.

Thank you again for helps.

Jason

On Tue, Jan 7, 2014 at 3:30 AM, joergprante@gmail.com <joergprante@gmail.com

wrote:

I was not speaking of the whole logfile, but the whole stacktrace, just to
make sure where the culprit started. If there was no OOM or other things
around, it would be surprising.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFi-T8iiKTULr_S-7OddJnAoGGF4%3Db6reB3vGejDArgnQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itw2EJvBU2wt1KeXg4xjuPVK6B-7UK8kJMtj56R1E3owcg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #6

Yes, it looks like two nodes do not agree about an update action and a
version conflict is pinging between them, node1 and node4.

Not sure if this happens while index recovery or while an update is
executed, but it is definitely worth raising an issue at the Elasticsearch
github to let the Elasticsearch core team have a look. It might be some
kind of a deadlock.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGKNB-1KXab4eWhDnKpe4szdPsidEWq2his2j%3DfPwU7Zw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jason Wee) #7

Jörg,

Done, https://github.com/elasticsearch/elasticsearch/issues/4639

Today when I investigated this issue, and just do a query to the time stamp
when the exceptions is happening, data were indexed though. The reason I
query is that, we worry if there is no data index during that period
exceptions are happening , thus data lost.

Thank you.

Jason

On Tue, Jan 7, 2014 at 4:34 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

Yes, it looks like two nodes do not agree about an update action and a
version conflict is pinging between them, node1 and node4.

Not sure if this happens while index recovery or while an update is
executed, but it is definitely worth raising an issue at the Elasticsearch
github to let the Elasticsearch core team have a look. It might be some
kind of a deadlock.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGKNB-1KXab4eWhDnKpe4szdPsidEWq2his2j%3DfPwU7Zw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itzoqWdujn713RK83ZZL4iGr19nY9nz34wbRtTKOSzcMNA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #8