Hi Guys,
I am using elasticsearch in a testing setup where the application asks for available users from pool of users in elasticsearch and maintains the state( free/occupied ).
The tests are run in multi thread environment, for example 5 java threads run in parallel to test the application
Each Java threads creates elasticsearch node and does search query and update query as the test goes on.
This issue with me is see a lot of queries failing when the test proceeds.
Some of the error/error stacks are as follows
[userindex3][0] received shard failed for [userindex3][0], node[5iSc3ZRMQ1-WLaQtDCh5zg], [R], s[INITIALIZING], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-05-29T05:54:10.669Z], details[shard failure [failed recovery][Recovery
Recovery failed from [3-D Man][Y-NXrbBSQZOHzEd7Jwx_rg][][inet[/IP] into [Oneg the Prober][5iSc3ZRMQ1-WLaQtDCh5zg][][inet[/ip:9301]]]; nested: RemoteTransportException[Failed to deserialize exception response from stream]
[2017-05-29 11:33:56,738][WARN ][gateway.local ] [3-D Man] [userindex3][4]: failed to list shard for shard_store on node [WAAT9EOsRSGK-QQ-17hi
org.elasticsearch.action.FailedNodeException: Failed node [WAAT9EOsRSGK-QQ-17hi4w]
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onFailure(TransportNodesOperationAction.java:210)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$1000(TransportNodesOperationAction.java:101)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleException(TransportNodesOperationAction.java:182)
at org.elasticsearch.transport.TransportService$3.run(TransportService.java:290)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:183)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Failed to obtain node lock, is the following location writable?:
Tested various elasticserch configurations with
A Cluster has 1/2/3/4 nodes
I always have one Index and one shard
Can somebody please tell me why this configuration is not able to sustain and fails?
Is there any standard configuration or formula when I set up nodes and try to connect them with Java client and avoid these errors.
The machine is 32GB with 8 cores which hosts the elasticsearch nodes and also runs the Java Threads which query elasticsearch.
For some reasons i am using elasticsearch 1.7.2
Any kind of help/Doc/link will be appreciated.
Thanks,
Ayush