Need Help --- How to Handle "org.elasticsearch.transport.ConnectTransportException" in Java Application

mohammad_rafi_g · August 7, 2013, 7:59am

Hi to All,
We have 4 nodes[1 Master Node+1 client Node + 2 Data Nodes] and
the settings of each node is:
each node heap size is 5gb["ES_MIN_MEM" and "ES_MAX_MEM" are
5gb]
and in elasticsearch.yml file
bootstrap.mlockall: true
index.store.compress.stored: true
index.refresh_interval: 20s
indices.memory.index_buffer_
size: 50%
index.translog.flush_threshold_ops: 50000
threadpool.bulk.type: fixed
threadpool.bulk.size: 90

     The 4 nodes are running under same cluster, and every thing is

working fine for bulk loading. But
When we are trying to query the elasticsearch data, while a bulk
loading happening in background, one node got down out of 4 nodes. and we
got the following Exception at java application log file.,
................................................................................................................................................................
05.08.2013 18:18:00 DEBUG (Log4jESLogger.java:99) - [Thing] failed to
connect to node [[#transport#-3][inet[/10.14.133.89:9313]]], removed from
nodes list
org.elasticsearch.transport.ConnectTransportException: [][inet[/
xxx.xxx.xxx.xxx:9304 http://10.14.133.89:9313]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:604)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:574)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:127)
at
org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:302)
at
org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:281)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: connection timed out
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:136)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:82)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
................................................................................................................................................................
And at elasticsearch logs we found the following...
at Data Node: "java.lang.OutOfMemoryError: Java heap space" and
" java.io.StreamCorruptedException"
at Master Node:
"org.elasticsearch.transport.NodeDisconnectedException",

"org.elasticsearch.indices.IndexMissingException: [20130730] missing" and

"org.elasticsearch.index.IndexShardMissingException: [20130727][6] missing"
................................................................................................................................................................

Now i want to handle this
"org.elasticsearch.transport.ConnectTransportException" Exception in my
java Application.
so i did the following..
//in java application this Exception is triggered by this Client[when node
is got down]object for every 5 seconds, so i added try-catch block for this
Client object.
Client client = null;
try{
//here we are assigning the TransportClient object[of data
nodes].
client =
getClient(EDRPropertyBean.getMapClients());
}catch(org.elasticsearch.transport.ConnectTransportException
cTE){
log.error("Some Exception message");
// here we have Exception Handle Logic
}
But This Exception Handling is not working, still we are getting the same
Exception instead of our Exception Handle Logic.

Please Tell me What I have to Do for Catch the
"org.elasticsearch.transport.ConnectTransportException" Exception
so that I will run my Exception Handle Logic to handle the
Node-Down[means, node is down]. and also
Please tell me if any other settings require for these nodes to handle
the both querying and bulk loading at a time.

Thanks and Regards
G Md Rafi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

spinscale · August 7, 2013, 8:26am

Hey,

the problem is not the ConnectException, but the OutOfMemoryError entry.
This means you fired a query which resulted in loading too much data into
the memory of the JVM.

The simplest solution is to give more memory to the JVM if you have it. The
more tedious solution is to analyze your query and see if you are doing
something wrong (like facetting on an analyzed field for example). Hard to
tell without more information.

Hope this helps

--Alex

On Wed, Aug 7, 2013 at 9:59 AM, rafi me.mrafi.en@gmail.com wrote:

Hi to All,
We have 4 nodes[1 Master Node+1 client Node + 2 Data Nodes] and
the settings of each node is:
each node heap size is 5gb["ES_MIN_MEM" and "ES_MAX_MEM"
are 5gb]
and in elasticsearch.yml file
bootstrap.mlockall: true
index.store.compress.stored: true
index.refresh_interval: 20s
indices.memory.index_buffer_
**size: 50%
index.translog.flush_**threshold_ops: 50000
threadpool.bulk.type: fixed
threadpool.bulk.size: 90
     The 4 nodes are running under same cluster, and every thing is
working fine for bulk loading. But
When we are trying to query the elasticsearch data, while a bulk
loading happening in background, one node got down out of 4 nodes. and we
got the following Exception at java application log file.,
............................................................
............................................................
..............................**..........
05.08.2013 18:18:00 DEBUG (Log4jESLogger.java:99) - [Thing] failed to
connect to node [[#transport#-3][inet[/10.14.**133.89:9313]]], removed
from nodes list
org.elasticsearch.transport.**ConnectTransportException: [inet[/
xxx.xxx.xxx.xxx:9304 http://10.14.133.89:9313]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.
connectToChannels(**NettyTransport.java:683)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.**java:604)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.**java:574)
at org.elasticsearch.transport.**TransportService.connectToNode(
TransportService.java:127)
at org.elasticsearch.client.**transport.**TransportClientNodesService$
**SimpleNodeSampler.sample(**TransportClientNodesService.**java:302)
at org.elasticsearch.client.**transport.**TransportClientNodesService$
**ScheduledNodeSampler.run(**TransportClientNodesService.**java:281)
at java.util.concurrent.**ThreadPoolExecutor.runWorker(**Unknown
Source)
at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: connection timed out
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientBoss.**processConnectTimeout(**NioClientBoss.java:136)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientBoss.process(**NioClientBoss.java:82)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(**AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.
NioClientBoss.run(**NioClientBoss.java:41)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
............................................................
............................................................
........................................
And at elasticsearch logs we found the following...
at Data Node: "java.lang.OutOfMemoryError: Java heap space" and
" java.io.**StreamCorruptedException"
at Master Node: "org.elasticsearch.transport.**NodeDisconnectedException",
                            "org.elasticsearch.indices.**IndexMissingException:
[20130730] missing" and
"org.elasticsearch.index.IndexShardMissingException:
[20130727][6] missing"
............................................................**
............................................................
..............................**..........

Now i want to handle this "org.elasticsearch.transport.ConnectTransportException"
Exception in my java Application.
so i did the following..
//in java application this Exception is triggered by this Client[when node
is got down]object for every 5 seconds, so i added try-catch block for this
Client object.
Client client = null;
try{
//here we are assigning the TransportClient object[of data
nodes].
client = getClient(EDRPropertyBean.
getMapClients()); **
}catch(org.elasticsearch.**transport.**ConnectTransportException
cTE){
log.error("Some Exception message");
// here we have Exception Handle Logic
}
But This Exception Handling is not working, still we are getting the same
Exception instead of our Exception Handle Logic.

Please Tell me What I have to Do for Catch the
"org.elasticsearch.transport.**ConnectTransportException" Exception
so that I will run my Exception Handle Logic to handle the
Node-Down[means, node is down]. and also

Please tell me if any other settings require for these nodes to handle
the both querying and bulk loading at a time.

Thanks and Regards
G Md Rafi

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

mohammad_rafi_g · August 7, 2013, 9:13am

Hi Alex.,

Thanks for Replying..

As per you Reply,

The more tedious solution is to analyze your query and see if you are
doing something wrong (like facetting on an analyzed field for example).
Hard to tell without more information.
What We have done is, we tested the load balance and
performance of the elasticsearch documents.
For this we used jmeter. we run this test with 100 threads for
300seconds.
We Queried all indices[20 indices with 500 million documets]. We ran
this jmeter query test when bulk index operation is going on.

Here we used simple query with routing. 
         like:

http://:///_search?q=:&routing=&pretty=true

     We need to handle the index operation with out any data loss when

any data node got down. So fro this,
by handling the "ConnectTransportException" or "ConnectException" in
java application[this java application loads the data into elasticsearch]
so that when ever a node got down we need to stop the indexing
operation.

     I have tried to catch these exceptions but i am not able to catch

these Exceptions.
1. Please tell me, if any one know, how to catch this Exceptions[
"ConnectTransportException" or "ConnectException"] in java application.

the problem is not the ConnectException, but the OutOfMemoryError entry.
This means you fired a query which resulted in loading too much data into
the memory of the JVM
Yes, the node got down due to OutofMemory Error. As of now,
we have 5gb for each node.
2. please tell me, for indexing and querying on huge data[say, above
100 million douments] what amount of jvm should i give to each node?[we
have 128gb of RAM]

Thanks and Regards
G Md Rafi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.