There are 6 indexes with 612 shards (each index 102 shards) and about
3,000,000,000 documents (1500 GB) clustered on 14 nodes.
During the last 2 days there were neither search nor indexing request
against ES. Today I checked the ES status, it was red and 4 nodes were not
accessible and I had to reset them.
The server hang occurs frequently (at least one server a day).
max_open_files is 1048555 (I see the number in the logs, during the ES
startup) and also bootstrap.mlockall: true configured in the .yml files.
ES version is 20.5
Linux version: 2.6.32-5-amd64 (Debian 2.6.32-46)
32g total ram (12g assigned to ES)
The hanging nodes are a problem no doubt. I can't really tell what causes
this from the infos you gave us but the number of shards you are using is
certainly way to high. Maybe you start with something like 14 shards or so.
This will also reduce the # open files dramatically. Yet, I'd recommend to
also move to a newer version of Elasticsearch (0.90).
simon
On Monday, August 19, 2013 11:20:39 AM UTC+2, Vahid wrote:
Hi all,
There are 6 indexes with 612 shards (each index 102 shards) and about
3,000,000,000 documents (1500 GB) clustered on 14 nodes.
During the last 2 days there were neither search nor indexing request
against ES. Today I checked the ES status, it was red and 4 nodes were not
accessible and I had to reset them.
The server hang occurs frequently (at least one server a day).
max_open_files is 1048555 (I see the number in the logs, during the ES
startup) and also bootstrap.mlockall: true configured in the .yml files.
ES version is 20.5
Linux version: 2.6.32-5-amd64 (Debian 2.6.32-46)
32g total ram (12g assigned to ES)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.