"No active shards" error


(Lenak) #1

Hi
We still working with elasticsearch 0.6.0.
Installation process of our system include “data population” task. One of the functions in the task is indexing. It is a small amount of documents for indexing ~100. But sometimes the population is stick (client node). There are two nodes in the system: one – elaticsearch server (index), second – java application with “no data”, “client” node.
Sometimes we see the following error in log of “client” node when index node is running

com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel com.hp.f8upper.services.tenant.client.rpc.itcase.ITCaseService.create(com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel)' threw an unexpected exception: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute [query] total failure; shardFailures {[null][1579430674][4]: No active shards}
at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:378)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:581)

What is a reason to be “No active” ?

Sometimes no errors in log (both sides) but task is stick . We print thread dump in the client side and got the following:

"http-8086-20" daemon prio=6 tid=0x3e9df400 nid=0x4640 waiting on condition [0x405ae000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0a62cfd8> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:73)
at org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:102)
at com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:170)
at com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:150)
at com.hp.f8upper.search.service.ElasticSearchServiceImpl.index(ElasticSearchServiceImpl.java:87)
at com.hp.f8upper.search.adapters.AbstractJsonAdapter.index(AbstractJsonAdapter.java:45)
at com.hp.f8upper.services.community.rpc.message.impl.MessageServiceImpl.create(MessageServiceImpl.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Any ideas what is a reason of stick?
Thanks
Lena


(Shay Banon) #2

Has the client found the other nodes?

On Wed, Jun 2, 2010 at 11:06 PM, Lenak lenakovganov@gmail.com wrote:

Hi
We still working with elasticsearch 0.6.0.
Installation process of our system include “data population” task. One of
the functions in the task is indexing. It is a small amount of documents
for indexing ~100. But sometimes the population is stick (client node).
There are two nodes in the system: one – elaticsearch server (index),
second
– java application with “no data”, “client” node.
Sometimes we see the following error in log of “client” node when index
node is running

com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public
abstract com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel

com.hp.f8upper.services.tenant.client.rpc.itcase.ITCaseService.create(com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel)'
threw an unexpected exception:
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute [query] total failure; shardFailures {[null][1579430674][4]: No
active shards}
at
com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:378)
at
com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:581)

What is a reason to be “No active” ?

Sometimes no errors in log (both sides) but task is stick . We print
thread dump in the client side and got the following:

"http-8086-20" daemon prio=6 tid=0x3e9df400 nid=0x4640 waiting on condition
[0x405ae000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0a62cfd8> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
at

org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:73)
at

org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:102)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:170)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:150)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.index(ElasticSearchServiceImpl.java:87)
at

com.hp.f8upper.search.adapters.AbstractJsonAdapter.index(AbstractJsonAdapter.java:45)
at

com.hp.f8upper.services.community.rpc.message.impl.MessageServiceImpl.create(MessageServiceImpl.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Any ideas what is a reason of stick?
Thanks
Lena

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/No-active-shards-error-tp865818p865818.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(Lenak) #3

There are two nodes in the system - one ES node (index) and second one -
client.
After start client node was connected to ES node with index (according to
log) .
Lena

On Sat, Jun 5, 2010 at 1:17 AM, kimchy [via ElasticSearch Users] <
ml-node+871683-1554071077-253818@n3.nabble.comml-node%2B871683-1554071077-253818@n3.nabble.com

wrote:

Has the client found the other nodes?

On Wed, Jun 2, 2010 at 11:06 PM, Lenak <[hidden email]http://user/SendEmail.jtp?type=node&node=871683&i=0

wrote:

Hi
We still working with elasticsearch 0.6.0.
Installation process of our system include “data population” task. One of
the functions in the task is indexing. It is a small amount of documents
for indexing ~100. But sometimes the population is stick (client node).
There are two nodes in the system: one – elaticsearch server (index),
second
– java application with “no data”, “client” node.
Sometimes we see the following error in log of “client” node when index
node is running

com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public
abstract com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel

com.hp.f8upper.services.tenant.client.rpc.itcase.ITCaseService.create(com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel)'
threw an unexpected exception:
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute [query] total failure; shardFailures {[null][1579430674][4]: No
active shards}
at
com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:378)
at
com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:581)

What is a reason to be “No active” ?

Sometimes no errors in log (both sides) but task is stick . We print
thread dump in the client side and got the following:

"http-8086-20" daemon prio=6 tid=0x3e9df400 nid=0x4640 waiting on
condition
[0x405ae000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0a62cfd8> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
at

org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:73)
at

org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:102)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:170)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:150)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.index(ElasticSearchServiceImpl.java:87)
at

com.hp.f8upper.search.adapters.AbstractJsonAdapter.index(AbstractJsonAdapter.java:45)
at

com.hp.f8upper.services.community.rpc.message.impl.MessageServiceImpl.create(MessageServiceImpl.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Any ideas what is a reason of stick?
Thanks
Lena

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/No-active-shards-error-tp865818p865818.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


View message @
http://elasticsearch-users.115913.n3.nabble.com/No-active-shards-error-tp865818p871683.html
To unsubscribe from "No active shards" error, click herehttp://elasticsearch-users.115913.n3.nabble.com/subscriptions/Unsubscribe.jtp?code=bGVuYWtvdmdhbm92QGdtYWlsLmNvbXw4NjU4MTh8MTMwNzY0OTAxMA==.


(Shay Banon) #4

I suggest, if you can, to move to 0.8 (which I think you have, based on
another thread ... ).

-shay.banon

On Sun, Jun 6, 2010 at 9:56 AM, Lenak lenakovganov@gmail.com wrote:

There are two nodes in the system - one ES node (index) and second one -
client.
After start client node was connected to ES node with index (according to
log) .
Lena

On Sat, Jun 5, 2010 at 1:17 AM, kimchy [via ElasticSearch Users] <[hidden
email] http://user/SendEmail.jtp?type=node&node=873759&i=0> wrote:

Has the client found the other nodes?

On Wed, Jun 2, 2010 at 11:06 PM, Lenak <[hidden email]http://user/SendEmail.jtp?type=node&node=871683&i=0

wrote:

Hi
We still working with elasticsearch 0.6.0.
Installation process of our system include “data population” task. One
of
the functions in the task is indexing. It is a small amount of documents
for indexing ~100. But sometimes the population is stick (client node).
There are two nodes in the system: one – elaticsearch server (index),
second
– java application with “no data”, “client” node.
Sometimes we see the following error in log of “client” node when index
node is running

com.google.gwt.user.server.rpc.UnexpectedException: Service method
'public
abstract com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel

com.hp.f8upper.services.tenant.client.rpc.itcase.ITCaseService.create(com.hp.f8upper.datamodel.client.tenant.IBaseCaseDataModel)'
threw an unexpected exception:
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute [query] total failure; shardFailures {[null][1579430674][4]: No
active shards}
at
com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:378)
at
com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:581)

What is a reason to be “No active” ?

Sometimes no errors in log (both sides) but task is stick . We print
thread dump in the client side and got the following:

"http-8086-20" daemon prio=6 tid=0x3e9df400 nid=0x4640 waiting on
condition
[0x405ae000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0a62cfd8> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
at

java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
at

org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:73)
at

org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:102)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:170)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.search(ElasticSearchServiceImpl.java:150)
at

com.hp.f8upper.search.service.ElasticSearchServiceImpl.index(ElasticSearchServiceImpl.java:87)
at

com.hp.f8upper.search.adapters.AbstractJsonAdapter.index(AbstractJsonAdapter.java:45)
at

com.hp.f8upper.services.community.rpc.message.impl.MessageServiceImpl.create(MessageServiceImpl.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Any ideas what is a reason of stick?
Thanks
Lena

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/No-active-shards-error-tp865818p865818.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


View message @
http://elasticsearch-users.115913.n3.nabble.com/No-active-shards-error-tp865818p871683.html
To unsubscribe from "No active shards" error, click here.


View this message in context: Re: "No active shards" errorhttp://elasticsearch-users.115913.n3.nabble.com/No-active-shards-error-tp865818p873759.html

Sent from the ElasticSearch Users mailing list archivehttp://elasticsearch-users.115913.n3.nabble.com/at Nabble.com.


(system) #5