I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster (application
is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But my
call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILA
BLE/2/no master];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at
org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:197)
at
org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:65)
at
org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:143)
at
org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(TransportAction.java:117)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by ES
server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in the
ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But my
call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILA
BLE/2/no master];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at
org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:197)
at
org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:65)
at
org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:143)
at
org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(TransportAction.java:117)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by ES
server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in the
ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But
my call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILA
BLE/2/no master];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at
org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:197)
at
org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:65)
at
org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:143)
at
org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(TransportAction.java:117)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by ES
server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in
the ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
Yes. This issue was opened at my request. It seems there are bugs in ES.
And this is a pretty big one in my opinion.
On Aug 4, 2014 12:38 PM, "Alexander Reelsen" alr@spinscale.de wrote:
I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But
my call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILA
BLE/2/no master];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at
org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:197)
at
org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:65)
at
org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:143)
at
org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(TransportAction.java:117)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by ES
server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in
the ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
By the way, is this bug seen with the TransportClient also, or just the
NodeClient?
Thanks!
Brian
On Monday, August 4, 2014 4:27:35 AM UTC-4, Alexander Reelsen wrote:
Hey,
Just a remote guess without knowing more: On your client side, the
exception is wrapped, so you need to unwrap it first.
--Alex
On Wed, Jul 23, 2014 at 9:47 AM, Cosmin-Radu Vasii <cosminra...@gmail.com
<javascript:>> wrote:
I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But
my call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILA
BLE/2/no master];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at
org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:197)
at
org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:65)
at
org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:143)
at
org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(TransportAction.java:117)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by ES
server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in
the ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But
my call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_
UNAVAILA
BLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.
globalBlockedException(ClusterBlocks.java:138)
at org.elasticsearch.cluster.block.ClusterBlocks.
globalBlockedRaiseException(ClusterBlocks.java:128)
at org.elasticsearch.action.bulk.TransportBulkAction.
executeBulk(TransportBulkAction.java:197)
at org.elasticsearch.action.bulk.TransportBulkAction.access$
000(TransportBulkAction.java:65)
at org.elasticsearch.action.bulk.TransportBulkAction$1.
onFailure(TransportBulkAction.java:143)
at org.elasticsearch.action.support.TransportAction$
ThreadedActionListener$2.run(TransportAction.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by ES
server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in
the ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
clusterState.blocks().globalBlockedRaiseException(
ClusterBlockLevel.WRITE);
....}
Is this a known situation or a known bug or I am missing something?
ES guys told they will fix this in the next release, probably in like two
months. We wanted to catch this exception so we could do a retry if ES
cluster is down(we have some sort of SLA which ensure the users that every
document will be indexed). We found two ways to fix it:
Do a request before the bulk insert to check if the cluster is ok - We
didn't do this, because it's an extra hop for every request and when the
cluster is ok it doesn't make sense to waste time querying for cluster
state.
We performed the action with a timeout and we consider that if the
timeout interval is reached the cluster is down. This is also not great,
because sometimes you might have some bottlenecks somewhere and you can
receive a timeout even though the cluster is up and running. So when
choosing the timeout period you have to be very carefull, maybe run some
stress/benchmarking tests before to see how fast you index. Just to be sure
we set it to 30s.
luni, 4 august 2014, 22:36:31 UTC+3, Brian a scris:
Alex,
By the way, is this bug seen with the TransportClient also, or just the
NodeClient?
Thanks!
Brian
On Monday, August 4, 2014 4:27:35 AM UTC-4, Alexander Reelsen wrote:
Hey,
Just a remote guess without knowing more: On your client side, the
exception is wrapped, so you need to unwrap it first.
I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But
my call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILA
BLE/2/no master];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at
org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:197)
at
org.elasticsearch.action.bulk.TransportBulkAction.access$000(TransportBulkAction.java:65)
at
org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(TransportBulkAction.java:143)
at
org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(TransportAction.java:117)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by ES
server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in
the ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
ES guys told they will fix this in the next release, probably in like two
months. We wanted to catch this exception so we could do a retry if ES
cluster is down(we have some sort of SLA which ensure the users that every
document will be indexed). We found two ways to fix it:
Do a request before the bulk insert to check if the cluster is ok - We
didn't do this, because it's an extra hop for every request and when the
cluster is ok it doesn't make sense to waste time querying for cluster
state.
We performed the action with a timeout and we consider that if the
timeout interval is reached the cluster is down. This is also not great,
because sometimes you might have some bottlenecks somewhere and you can
receive a timeout even though the cluster is up and running. So when
choosing the timeout period you have to be very carefull, maybe run some
stress/benchmarking tests before to see how fast you index. Just to be sure
we set it to 30s.
luni, 4 august 2014, 22:36:31 UTC+3, Brian a scris:
Alex,
By the way, is this bug seen with the TransportClient also, or just the
NodeClient?
Thanks!
Brian
On Monday, August 4, 2014 4:27:35 AM UTC-4, Alexander Reelsen wrote:
Hey,
Just a remote guess without knowing more: On your client side, the
exception is wrapped, so you need to unwrap it first.
I am using the dataless NodeClient to connect to my cluster (version is
1.1.1). Everything is working ok, except when failures occur. The scenario
is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal. But
my call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_
UNAVAILA
BLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.
globalBlockedException(ClusterBlocks.java:138)
at org.elasticsearch.cluster.block.ClusterBlocks.
globalBlockedRaiseException(ClusterBlocks.java:128)
at org.elasticsearch.action.bulk.TransportBulkAction.
executeBulk(TransportBulkAction.java:197)
at org.elasticsearch.action.bulk.TransportBulkAction.access$
000(TransportBulkAction.java:65)
at org.elasticsearch.action.bulk.TransportBulkAction$1.
onFailure(TransportBulkAction.java:143)
at org.elasticsearch.action.support.TransportAction$
ThreadedActionListener$2.run(TransportAction.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by
ES server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in
the ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
clusterState.blocks().globalBlockedRaiseException(
ClusterBlockLevel.WRITE);
....}
Is this a known situation or a known bug or I am missing something?
I don't like that much the TransportClient. I prefer the NodeClient,
because is cluster aware, it doesn't have to do a double hop when sending
requests, it will directly call the corresponding data node. I know that
the node client does also balancing, which I don't think is the case for
the TransportClient. Please correct me if I'm wrong. Anyway, the bug here
is a real bug and the result for ClusterBlockException should be the same
for both Node and Transport Client.
ES guys told they will fix this in the next release, probably in like two
months. We wanted to catch this exception so we could do a retry if ES
cluster is down(we have some sort of SLA which ensure the users that every
document will be indexed). We found two ways to fix it:
Do a request before the bulk insert to check if the cluster is ok - We
didn't do this, because it's an extra hop for every request and when the
cluster is ok it doesn't make sense to waste time querying for cluster
state.
We performed the action with a timeout and we consider that if the
timeout interval is reached the cluster is down. This is also not great,
because sometimes you might have some bottlenecks somewhere and you can
receive a timeout even though the cluster is up and running. So when
choosing the timeout period you have to be very carefull, maybe run some
stress/benchmarking tests before to see how fast you index. Just to be sure
we set it to 30s.
luni, 4 august 2014, 22:36:31 UTC+3, Brian a scris:
Alex,
By the way, is this bug seen with the TransportClient also, or just the
NodeClient?
Thanks!
Brian
On Monday, August 4, 2014 4:27:35 AM UTC-4, Alexander Reelsen wrote:
Hey,
Just a remote guess without knowing more: On your client side, the
exception is wrapped, so you need to unwrap it first.
I am using the dataless NodeClient to connect to my cluster (version
is 1.1.1). Everything is working ok, except when failures occur. The
scenario is the following:
-I have an application java based which connects to ES Cluster
(application is started and the cluster is up and running)
-I shutdown the cluster
-I try to send a bulk request
-The following exception is displayed in the logs, which is normal.
But my call never catches the exception:
Exception in thread "elasticsearch[Lasher][generic][T#6]"
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_
UNAVAILA
BLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.
globalBlockedException(ClusterBlocks.java:138)
at org.elasticsearch.cluster.block.ClusterBlocks.
globalBlockedRaiseException(ClusterBlocks.java:128)
at org.elasticsearch.action.bulk.TransportBulkAction.
executeBulk(TransportBulkAction.java:197)
at org.elasticsearch.action.bulk.TransportBulkAction.access$
000(TransportBulkAction.java:65)
at org.elasticsearch.action.bulk.TransportBulkAction$1.
onFailure(TransportBulkAction.java:143)
at org.elasticsearch.action.support.TransportAction$
ThreadedActionListener$2.run(TransportAction.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
My code is something like this
BulkResponse response;
try {
response = requestBuilder.execute().actionGet();
}
catch(NoNodeAvailableException ex){
LOGGER.error("Cannot connect to ES Cluster: " +
LOGGER.error("Exception in processing indexing request by
ES server. " + ex.getMessage());
}
When I use a single request everything is ok. I also noticed a TODO in
the ES code in the TransportBulkAction.java
private void executeBulk(final BulkRequest bulkRequest, final long
startTime, final ActionListener listener, final
AtomicArray responses ) {
ClusterState clusterState = clusterService.state();
// TODO use timeout to wait here if its blocked...
clusterState.blocks().globalBlockedRaiseException(
ClusterBlockLevel.WRITE);
....}
Is this a known situation or a known bug or I am missing something?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.