Idle Elasticsearch causing servers to hang

Elastic Stack Elasticsearch
#1

Hi all,

There are 6 indexes with 612 shards (each index 102 shards) and about
3,000,000,000 documents (1500 GB) clustered on 14 nodes.
During the last 2 days there were neither search nor indexing request
against ES. Today I checked the ES status, it was red and 4 nodes were not
accessible and I had to reset them.

The server hang occurs frequently (at least one server a day).
max_open_files is 1048555 (I see the number in the logs, during the ES
startup) and also bootstrap.mlockall: true configured in the .yml files.

ES version is 20.5
Linux version: 2.6.32-5-amd64 (Debian 2.6.32-46)
32g total ram (12g assigned to ES)

Thanks for your helps,
Vahid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

#2

The hanging nodes are a problem no doubt. I can't really tell what causes
this from the infos you gave us but the number of shards you are using is
certainly way to high. Maybe you start with something like 14 shards or so.
This will also reduce the # open files dramatically. Yet, I'd recommend to
also move to a newer version of Elasticsearch (0.90).

simon

On Monday, August 19, 2013 11:20:39 AM UTC+2, Vahid wrote:

Hi all,

There are 6 indexes with 612 shards (each index 102 shards) and about
3,000,000,000 documents (1500 GB) clustered on 14 nodes.
During the last 2 days there were neither search nor indexing request
against ES. Today I checked the ES status, it was red and 4 nodes were not
accessible and I had to reset them.

The server hang occurs frequently (at least one server a day).
max_open_files is 1048555 (I see the number in the logs, during the ES
startup) and also bootstrap.mlockall: true configured in the .yml files.

ES version is 20.5
Linux version: 2.6.32-5-amd64 (Debian 2.6.32-46)
32g total ram (12g assigned to ES)

Thanks for your helps,
Vahid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

#3