Workaround when ES hangs on stale NFS mount?

Elastic Stack Elasticsearch
1

Hi all,

it is a known issue that ES hangs on startup for a couple of minutes when there is a stale NFS mount on the system: https://github.com/elastic/elasticsearch/pull/24402

This is particularly a problem as also the client hangs when PreBuildTransportClient is used in our webapp.

So i wonder how to fix this issue. The following options come to my mind:

  • We "fix" this directly by modifying source code in org.apache.lucene.util.IOUtils
  • We use ES 6.0.0-alpha2 in which this issue is solved AFAIK

Is there any other way to get rid of this issue?

Can 6.0.0-alpha2 be used in a productive environment as a single node cluster with quite simple queries running? When is 6.0.0 released?

Any help would be appreciated...

Regards,

Abid

2

I "solved" the issue by placing a hack in org.apache.lucene.util.IOUtils

  • Download sources from https://github.com/apache/lucene-solr (switch to the tagged release branch according to the lucence-core version)
  • Modify org.apache.lucene.util.IOUtils.spins(Path) so that it always returns true (we are not using SSDs)
  • Build lucene-core and replace the old one in classpath ($ELASTICSEARCH_HOME$/lib with the modified one

This seems to work.

3

I unfortunately have to say that this did not solve the issue in the end.

As mentioned in https://github.com/elastic/elasticsearch/issues/24390, the problem is that ES iterates over all file stores on startup. So I do not have an idea how to fix this apart from waiting for release 6.0.0.

If there is any known workaround please let me know.

Avoiding stale nfs mounts is NOT possible in our environment.

4

Finally, it seems that we could at least avoid the hangup behaviour at client side.

We did this by modifiying the static initializer in org.elasticsearch.env.Environment. Instead of iterating on all filestores it simply creates an array containing only one filestore - the one behind the current directory. We assume that on client side the file store has no relevance as there is no data to be stored.

Just for curiosity I would like to know, what for ES needs all the file stores. E.g, when the data directory is located elsewhere than the installation directory?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.