Nexus 3 on docker on nfs4 (efs) reports highwatermark


(Luis) #1

Hi,

I installed nexus3 opensource (which uses ES 2.2) using the sonatype docker container. The underlying filesystem is nfsv4 (Amazon EFS) mounted at /nexus-data on the container.

When I start the nexus container, elasticsearch complains with an error:
*SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [....] high disk watermark [90%] exceeded on [.....][/nexus-data/elasticsearch/nexus/nodes/0] free: 8191.9pb[-99.9%], shards will be relocated away from this node

Both the host OS and the container correctly report disk space, but ES interprets it as a negative value (some kind of overflow?).

If I disable watermark checks with "cluster.routing.allocation.disk.threshold_enabled: false" I get no more errors and es and nexus work fine, but I wonder what else might break if ES disk information is broken.

See the details below. Any help is appreciated.

Disk space reported inside container

$ df -t nfs4
Filesystem 1K-blocks Used Available Use% Mounted on
REDACTED_EFS_ENDPOINT:/ 9007199254740992 107520 9007199254633472 1% /nexus-data

Cutdown report of _node/stats

"fs": {
"timestamp": 1475748403553,
"total": {
"total_in_bytes": **-**9223372036854775808,
"free_in_bytes": 9223372036744675328,
"available_in_bytes": 9223372036744675328,
"spins": "true"
},
"data": [{
"path": "/nexus-data/elasticsearch/nexus/nodes/0",
"mount": "/nexus-data (eu-west-1c.fs-276294ee.efs.eu-west-1.amazonaws.com:/)",
"type": "nfs4",
"total_in_bytes": **-**9223372036854775808,
"free_in_bytes": 9223372036744675328,
"available_in_bytes": 9223372036744675328,
"spins": "true"
}]
},

Elasticsearch details

{
"name" : "....",
"cluster_name" : "nexus",
"version" : {
"number" : "2.2.0",
"build_hash" : "NA",
"build_timestamp" : "NA",
"build_snapshot" : false,
"lucene_version" : "5.4.1"
},
"tagline" : "You Know, for Search"
}


(Yannick Welsch) #2

seems to be a duplicate of Elasticsearch with Amazon Elastic File System

It is fixed in 2.4.1 https://github.com/elastic/elasticsearch/pull/20527 and caused by a bug in the JDK: https://bugs.openjdk.java.net/browse/JDK-8162520:

Can you verify that 2.4.1 works for you?


(Luis) #3

Thank you, they are indeed the same.

I can't use ES 2.4.1 since the ES OSGI bundle is created by sonatype and also dependant on apache.servicemix to create a bundle for ES 2.4.1.

I've opened a bug with Sonatype to see if they can upgrade to 2.4.1 with their next release.

Thanks for the swift response!


(system) #4