Elaticsearch Version : 1.7.3
We hava 3 nodes in production.
Every node the java heap size is 31G and now it only use 10G.
Field cache is nearly 6G and we don't use DocValue.
The doc count is 42 billion.
We hava two indices.
The one has 64 primary shard and 2 replica shard , the another one has 16 primary shard and 2 replica shard.
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
Elasticsearch became unresponsive at March 4, but the Elasticsearch process was alive yet.
The state of 9300 port was also LISTEN.
There were many slow search at index-search-slowlog.log
-- [2016-03-04 19:26:49,404][WARN ][index.search.slowlog.query] [Kylun] [global][31]
took[7.8s], took_millis[7849], types[], stats[], search_type[QUERY_AND_FETCH],
total_shards[1], source[{"from":0,"size":21,"timeout":20000,"fields":["timestamp","rawMessage"],
"sort":[{"timestamp":{"order":"desc"}}]}], extra_source[],
And then Elasticsearch became Zombie after executing "jstack" and "jmap" command.
The "kill" command was not working. Only used "kill -9".
Now we restart Elasticsearch and it is normal now.
Anyone can help me?
Thanks.