We are using ElasticSearch to index documents that are stored in HBase, and
so I have set up ElasticSearch to use the hadoop gateway pointing to our
I've been running load tests on the system and find that after a day or
two, ES will hit its open file limit (presently set to 64,000). I've looked
at the output of 'sudo lsof -u elasticsearch' and there are many thousands
of open TCP connections in the CLOSE_WAIT state:
java 14109 elasticsearch 661u IPv6 9820568 0t0
TCP :38257->:50010 (CLOSE_WAIT)
I'm just starting to look into the problem more closely, but I found there
was a previous mailing list discussion about essentially the same issue:
There wasn't any resolution, so I thought I'd ask here if anyone has seen
or resolved this problem since.
Some more pertinent information:
- ElasticSearch version is 0.19.9
- We use the Cloudera hadoop distribution rather than the Apache
distribution. As a result, we cannot use the hadoop-core jar included with
the elasticsearch-hadoop plugin. I have worked around the problem by
prepending the locations of the correct jar files to ES_CLASSPATH in
elasticsearch.in.sh. I have a feeling this may be related to the problem,
but that's nothing more than a vague hunch.
Anyway, if anyone has seen similar problems or has suggestions for
debugging strategies, let me know.