I am upgrading from 0.17.x to 0.19.11. Have deleted old index files.
Stand alone jvm Elastic Search server starts fine. It connects to
standalone jvm client node and indexes data without any problem.
A 0.19.11 client node inside solaris weblogic web app connecting to the
above server node lands up creating hundreds of tranport_client_worker
threads and crashes weblogic.
I see hundreds of these lines in the crash log:
"0x00000001040d1000 JavaThread "elasticsearch[Oneg the
Prober][transport_server_worker][T#255]{New I/O worker #511}" daemon
[_thread_in_native, id=797, stack(0xfffffffe9f400000,0xfffffffe9f500000)]"
followed by hundreds of these:
"0x0000000103def000 JavaThread "ExecuteThread: '126' for queue:
'weblogic.socket.Muxer'" daemon [_thread_blocked, id=272,
stack(0xfffffffee2400000,0xfffffffee2500000)]"
My weblogic log shows below:
[INFO ][10-Dec 10:00:10,717][][node][Oneg the Prober] {0.19.11}[13351]:
starting ...
[INFO ][10-Dec 10:00:12,984][][transport][Oneg the Prober] bound_address
{inet[/169.49.110.160:9403]}, publish_address {inet[/169.49.110.160:9403]}
An unexpected error has been detected by Java Runtime Environment:
SIGBUS (0xa) at pc=0xffffffff7e2fe008, pid=13351, tid=560
Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode
solaris-sparc)
Problematic frame:
V [libjvm.so+0x6fe008]
An error report file with more information is saved as:
/app/securities/rapport/bin/hs_err_pid13351.log
If you would like to submit a bug report, please visit:
In point (1) below, I am also using "compress.lzf.decoder: safe" on the
standalone ES server
On Monday, 10 December 2012 15:29:24 UTC, ABC wrote:
Hi,
I am upgrading from 0.17.x to 0.19.11. Have deleted old index files.
Stand alone jvm Elastic Search server starts fine. It connects to
standalone jvm client node and indexes data without any problem.
A 0.19.11 client node inside solaris weblogic web app connecting to the
above server node lands up creating hundreds of tranport_client_worker
threads and crashes weblogic.
I see hundreds of these lines in the crash log:
"0x00000001040d1000 JavaThread "elasticsearch[Oneg the
Prober][transport_server_worker][T#255]{New I/O worker #511}" daemon
[_thread_in_native, id=797, stack(0xfffffffe9f400000,0xfffffffe9f500000)]"
followed by hundreds of these:
"0x0000000103def000 JavaThread "ExecuteThread: '126' for queue:
'weblogic.socket.Muxer'" daemon [_thread_blocked, id=272,
stack(0xfffffffee2400000,0xfffffffee2500000)]"
My weblogic log shows below:
[INFO ][10-Dec 10:00:10,717][node][Oneg the Prober] {0.19.11}[13351]:
starting ...
[INFO ][10-Dec 10:00:12,984][transport][Oneg the Prober] bound_address
{inet[/169.49.110.160:9403]}, publish_address {inet[/169.49.110.160:9403
]}
An unexpected error has been detected by Java Runtime Environment:
SIGBUS (0xa) at pc=0xffffffff7e2fe008, pid=13351, tid=560
Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode
solaris-sparc)
Problematic frame:
V [libjvm.so+0x6fe008]
An error report file with more information is saved as:
/app/securities/rapport/bin/hs_err_pid13351.log
If you would like to submit a bug report, please visit:
Which Java do you ran on ES server, which on ES client side?
JVM 10.0-b22 is Java 1.6.0_06 (April 2008), more than four years old. Such
old java versions have many bugs that will stop you from running ES
successfully.
Please update to the latest Java 7. Note that Java 6 is scheduled end of
life for February, 2013, by Oracle.
Be aware, you can not mix Java 6 and Java 7 in ES client/server
configurations due to JVM object serialization issues.
If you are bound to an obsolete Java version, you could try to tweak the
JVM parameters so they do not trigger subtle bugs, but YMMV. It will be
hard and frustrating.
There are many transport client threads, as thread pools have been enlarged
in ES since 0.17. Hundreds of threads are way too much of course. I assume
you have many transport client instances open or the transport client has
difficulties to connect and starts threads in panic while retrying. But,
it's a secondary error, the primary error is you can't connect at all.
I will try to get the jvm upgraded. Is it possible to limit the number of
threads the client node will generate while trying to connect to server? I
am assuming the thread pool settingshttp://www.elasticsearch.org/guide/reference/modules/threadpool.html are
only for the server nodes and not for client nodes. Please tell me if I am
wrong.
On Monday, 10 December 2012 17:24:22 UTC, Jörg Prante wrote:
Which Java do you ran on ES server, which on ES client side?
JVM 10.0-b22 is Java 1.6.0_06 (April 2008), more than four years old. Such
old java versions have many bugs that will stop you from running ES
successfully.
Please update to the latest Java 7. Note that Java 6 is scheduled end of
life for February, 2013, by Oracle.
Be aware, you can not mix Java 6 and Java 7 in ES client/server
configurations due to JVM object serialization issues.
If you are bound to an obsolete Java version, you could try to tweak the
JVM parameters so they do not trigger subtle bugs, but YMMV. It will be
hard and frustrating.
There are many transport client threads, as thread pools have been
enlarged in ES since 0.17. Hundreds of threads are way too much of course.
I assume you have many transport client instances open or the transport
client has difficulties to connect and starts threads in panic while
retrying. But, it's a secondary error, the primary error is you can't
connect at all.
Is there a way to limit the number of threads used by client node? I am
assuming the thread pool settingshttp://www.elasticsearch.org/guide/reference/modules/threadpool.html are
only for the server nodes and not for client nodes.
another observation is that the client node connects to the server and then
immediately fails. The ES server node logs show:
On Wednesday, 12 December 2012 11:53:13 UTC, ABC wrote:
Thanks Jorg,
I will try to get the jvm upgraded. Is it possible to limit the number of
threads the client node will generate while trying to connect to server? I
am assuming the thread pool settingshttp://www.elasticsearch.org/guide/reference/modules/threadpool.html are
only for the server nodes and not for client nodes. Please tell me if I am
wrong.
On Monday, 10 December 2012 17:24:22 UTC, Jörg Prante wrote:
Which Java do you ran on ES server, which on ES client side?
JVM 10.0-b22 is Java 1.6.0_06 (April 2008), more than four years old.
Such old java versions have many bugs that will stop you from running ES
successfully.
Please update to the latest Java 7. Note that Java 6 is scheduled end of
life for February, 2013, by Oracle.
Be aware, you can not mix Java 6 and Java 7 in ES client/server
configurations due to JVM object serialization issues.
If you are bound to an obsolete Java version, you could try to tweak the
JVM parameters so they do not trigger subtle bugs, but YMMV. It will be
hard and frustrating.
There are many transport client threads, as thread pools have been
enlarged in ES since 0.17. Hundreds of threads are way too much of course.
I assume you have many transport client instances open or the transport
client has difficulties to connect and starts threads in panic while
retrying. But, it's a secondary error, the primary error is you can't
connect at all.
Server and client nodes share the threadpool settings, for simplicity of
code design. A TransportClient does not use all thread pool types, only a
few.
You can limit the thread pools for a TransportClient that is only indexing
data like this. Assume you want ten Netty connections and ten thread for
index or bulk:
get both client and server nodes to run on the same jvm
Add "compress.lzf.decoder: safe" setting to both client and server.
Without both of the above, the client continued to crash.
Without the above 2 settings the solaris client node uses and blocks all
available threads. Surely this must be a bug that needs to be addressed. I
will add the client thread settings you mentioned as a safety net.
Thank you! Your suggestions were really useful!
On Wednesday, 12 December 2012 13:13:43 UTC, Jörg Prante wrote:
Hi,
Server and client nodes share the threadpool settings, for simplicity of
code design. A TransportClient does not use all thread pool types, only a
few.
You can limit the thread pools for a TransportClient that is only indexing
data like this. Assume you want ten Netty connections and ten thread for
index or bulk:
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.