Nexus 3 on docker on nfs4 (efs) reports highwatermark


I installed nexus3 opensource (which uses ES 2.2) using the sonatype docker container. The underlying filesystem is nfsv4 (Amazon EFS) mounted at /nexus-data on the container.

When I start the nexus container, elasticsearch complains with an error:
*SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [....] high disk watermark [90%] exceeded on [.....][/nexus-data/elasticsearch/nexus/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node

Both the host OS and the container correctly report disk space, but ES interprets it as a negative value (some kind of overflow?).

If I disable watermark checks with "cluster.routing.allocation.disk.threshold_enabled: false" I get no more errors and es and nexus work fine, but I wonder what else might break if ES disk information is broken.

See the details below. Any help is appreciated.

Disk space reported inside container

$ df -t nfs4
Filesystem 1K-blocks Used Available Use% Mounted on
REDACTED_EFS_ENDPOINT:/ 9007199254740992 107520 9007199254633472 1% /nexus-data

Cutdown report of _node/stats

"fs": {
"timestamp": 1475748403553,
"total": {
"total_in_bytes": **-**9223372036854775808,
"free_in_bytes": 9223372036744675328,
"available_in_bytes": 9223372036744675328,
"spins": "true"
"data": [{
"path": "/nexus-data/elasticsearch/nexus/nodes/0",
"mount": "/nexus-data (",
"type": "nfs4",
"total_in_bytes": **-**9223372036854775808,
"free_in_bytes": 9223372036744675328,
"available_in_bytes": 9223372036744675328,
"spins": "true"

Elasticsearch details

"name" : "....",
"cluster_name" : "nexus",
"version" : {
"number" : "2.2.0",
"build_hash" : "NA",
"build_timestamp" : "NA",
"build_snapshot" : false,
"lucene_version" : "5.4.1"
"tagline" : "You Know, for Search"

seems to be a duplicate of Elasticsearch with Amazon Elastic File System

It is fixed in 2.4.1 and caused by a bug in the JDK:

Can you verify that 2.4.1 works for you?

Thank you, they are indeed the same.

I can't use ES 2.4.1 since the ES OSGI bundle is created by sonatype and also dependant on apache.servicemix to create a bundle for ES 2.4.1.

I've opened a bug with Sonatype to see if they can upgrade to 2.4.1 with their next release.

Thanks for the swift response!