Hi together,
I just tested with lots of indices, updates and so on.
Everything ran great. Then suddenly the ClusterState went to red,
connections on my application timed out.
I searched through the log and interpreted some hints, but there is no real
error, why some shards got unassigned and so turned the ClusterState to red.
Index creation: 2 Shards, 1 Repica each. 2 Servers, 45GB heap each.
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1932,
"active_shards" : 3864,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 2
}
Even worse: Many many threads in my application stuck at one point:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f2b03c60ae8> (a org.elasticsearch.common.util.concurrent.BaseFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:271)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:113)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:45)
at sun.reflect.GeneratedMethodAccessor1210.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
The mentioned threads all got triggered by this call, where indexName is a
local parameter: IndicesExistsResponse existsResponse =
client.admin().indices().prepareExists(indexName).execute().actionGet()
Should'nt this return immediately?
Some further information and the hints I gathered from the logs:
Log snipped, where last_flush_time is mentioned. Is it right, that in this
example it takes to long for flushing the indices?
[2012-08-17 10:57:54,852][TRACE][index.translog ] [es_server_1]
[index_A][1] flushing translog, last_flush_time [1345192072951], breached
[30m]
[2012-08-17 10:57:54,852][TRACE][index.translog ] [es_server_1]
[index_A][0] flushing translog, last_flush_time [1345192073057], breached
[30m]
[2012-08-17 10:57:54,852][TRACE][index.shard.service ] [es_server_1]
[index_A][1] flush with full[false], refresh[false], force[false]
[2012-08-17 10:57:54,852][TRACE][index.shard.service ] [es_server_1]
[index_A][0] flush with full[false], refresh[false], force[false]
[2012-08-17 10:57:59,845][TRACE][index.translog ] [es_server_1]
[index_B][0] flushing translog, last_flush_time [1345192076266], breached
[30m]
[2012-08-17 10:57:59,845][TRACE][index.shard.service ] [es_server_1]
[index_B][0] flush with full[false], refresh[false], force[false]
[2012-08-17 10:57:59,845][TRACE][index.translog ] [es_server_1]
[index_C][1] flushing translog, last_flush_time [1345192076266], breached
[30m]
[2012-08-17 10:57:59,845][TRACE][index.shard.service ] [es_server_1]
[index_C];de][1] flush with full[false], refresh[false], force[false]
[2012-08-17 10:57:59,847][TRACE][index.translog ] [es_server_1]
[index_D][1] flushing translog, last_flush_time [1345192076266], breached
[30m]
[2012-08-17 10:57:59,847][TRACE][index.shard.service ] [es_server_1]
[index_D][1] flush with full[false], refresh[false], force[false]
[2012-08-17 10:57:59,849][TRACE][index.translog ] [es_server_1]
[index_D][0] flushing translog, last_flush_time [1345192076266], breached
[30m]
Timeout Exception in my application. I don't know what triggered it or what
was tried in background, because the lines are surrounded by others,
nothing to do with the search part.
2012-08-19 10:59:57: [org.elasticsearch.client.transport] - [Ent] failed to
get node info for [#transport#-2][inet[/192.168.1.123:9300]],
disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/
192.168.1.123:9300]][cluster/nodes/info] request_id [24067] timed out after
[5000ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:347)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
If you need further information, please let me know.
I appreciate any help I can get 
Thanks in advance
Greetings
Maik
--