# Elasticsearch with Amazon Elastic File System **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2016, 11:18am UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867 "2016-07-19T11:18:44Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![ivten](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@ivten](https://discuss.elastic.co/u/ivten)\ **Post date:** [July 19, 2016, 11:18am UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/1 "2016-07-19T11:18:44Z")
Hi, I have a single Elasticsearch node on Amazon EC2 instance. I've also mounted AWS Elastic File System to the instance under /usr/share/elasticsearch. When I start elasticsearch it starts normally and I can see that it creates files on the mounted EFS, but when I try to index a document I get this error: `{ "error" : { "root_cause" : [{ "type" : "unavailable_shards_exception", "reason" : "[customer][3] primary shard is not active Timeout: [1m], request: [index {[customer][external][1], source[\n{\n \"name\": \"John Doe\"\n}]}]" } ], "type" : "unavailable_shards_exception", "reason" : "[customer][3] primary shard is not active Timeout: [1m], request: [index {[customer][external][1], source[\n{\n \"name\": \"John Doe\"\n}]}]" }, "status" : 503 }` When I look at the elasticsearch output I see this error: `[2016-07-19 10:19:34,241][INFO][cluster.metadata] [Zarek] [customer] creating index, cause [auto(index api)], templates [], shards [5]/[1], mappings [external] [2016-07-19 10:19:34,348][WARN][indices.cluster] [Zarek] [[customer][1]] marking and sending shard failed due to [failed to create shard] java.lang.NullPointerException at org.elasticsearch.index.shard.ShardPath.selectNewPathForShard(ShardPath.java:241) at org.elasticsearch.index.IndexService.createShard(IndexService.java:336) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:601) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:501) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:166) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2016-07-19 10:19:34,351][WARN][cluster.action.shard] [Zarek] [customer][1] received shard failed for target shard [[customer][1], node[35FNNM0hSsCWOyRQZPsz8A], [P], v[1], s[INITIALIZING], a[id=piu83IRBQNCpi2OLm4aGrw], unassigned_info[[reason=INDEX_CREATED], at[2016-07-19T10:19:34.252Z]]], indexUUID [ii2mnw8tQaCrfQ27Z5tkvA], message [failed to create shard], failure [NullPointerException[null]] java.lang.NullPointerException ... [2016-07-19 10:19:34,364][WARN][indices.cluster] [Zarek] [[customer][3]] marking and sending shard failed due to [failed to create shard] java.lang.NullPointerException` The cluster health shows this: `{ "cluster_name" : "elasticsearch", "status" : "red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 10, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 0.0 }` Clearly elasticsearch cannot create shards on the Elastic File System. However at the same time it is able to use it as it stores a bunch of files and folders on it (like /usr/share/elasticsearch/nodes/0/indices and /usr/share/elasticsearch/nodes/0/\_state). I added rwx permissions to everyone under the mounted /usr/share/elasticsearch, but it doesn't help. Selinux is also disabled. Forcing shards reallocation doesn't help either. Am I missing something in regard to Elasticsearch and NFS permissions? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 19, 2016, 11:40am UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/2 "2016-07-19T11:40:42Z")
Don't use elasticsearch with NFS. ---
**Author:** ![ivten](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@ivten](https://discuss.elastic.co/u/ivten)\ **Post date:** [July 21, 2016, 2:36pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/3 "2016-07-21T14:36:48Z")
I am still testing elasticsearch, so performance isn't important for me at this stage. However I'm interested in resolving the error that I got, because it doesn't make any sense: elasticsearch writes files on the storage, but cannot save shards. ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [July 21, 2016, 3:51pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/4 "2016-07-21T15:51:31Z")
Hmm, is this ES 2.3? That line is here: > So it seems `bestPath` is null, but it seems like that should only happen if `env.nodePaths()` returned an empty array, but then when I look at `NodeEnvironment.java` and `Environtment.java` I don't see how an empty `NodePath` array can be set on `NodeEnvironment`... odd. Mike ---
**Author:** ![ivten](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@ivten](https://discuss.elastic.co/u/ivten)\ **Post date:** [July 21, 2016, 7:40pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/5 "2016-07-21T19:40:18Z")
Yes, it's the latest elasticsearch. I thought at first that it had something to do with ES not being able to acquire lock on the files on the AWS Elastic Files system. However the AWS documentation says that EFS does support file locks. So I guess there is some kind of NFS particularity that is bugging things. I also tried mounting the NFS with various options such as noatime,nolock,auto,sec=sys but without any luck. When I point the data directory to the local file system it works just fine. ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [July 22, 2016, 12:08pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/6 "2016-07-22T12:08:57Z")
OK I think I see one way that `NPE` could happen: if `FileStore.getUsableSpace()` returns `Long.MIN_VALUE` (which is odd!). I'll fix `ShardPath` to guard against this... Mike ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [July 22, 2016, 7:06pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/7 "2016-07-22T19:06:37Z")
I opened this PR: [https://github.com/elastic/elasticsearch/pull/19554](https://github.com/elastic/elasticsearch/pull/19554) ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [July 23, 2016, 8:12pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/8 "2016-07-23T20:12:56Z")
Hi @ivten, can you run `df` on this node and post the output here? We are trying to dig into the JDK sources to understand whether/how `FileStore.getUsableSpace` could return this. Thanks. ---
**Author:** ![ivten](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@ivten](https://discuss.elastic.co/u/ivten)\ **Post date:** [July 25, 2016, 7:56pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/9 "2016-07-25T19:56:10Z")
Here it is: `[ec2-user@ip-172-21-2-107 ~]$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 20509288 6005280 14403760 30% / devtmpfs 243056 60 242996 1% /dev tmpfs 251632 0 251632 0% /dev/shm eu-west-1a.fs-4514fa8c.efs.eu-west-1.amazonaws.com:/ 9007199254740992 0 9007199254740992 0% /usr/share/elasticsearch/data [ec2-user@ip-172-21-2-107 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 20G 5.8G 14G 30% / devtmpfs 238M 60K 238M 1% /dev tmpfs 246M 0 246M 0% /dev/shm eu-west-1a.fs-4514fa8c.efs.eu-west-1.amazonaws.com:/ 8.0E 0 8.0E 0% /usr/share/elasticsearch/data` ---
**Author:** ![rmuir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmuir/32/44949_2.png) [@rmuir](https://discuss.elastic.co/u/rmuir)\ **Post date:** [July 25, 2016, 10:09pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/10 "2016-07-25T22:09:38Z")
That is definitely the case. Thank you for confirming what is happening. I reported it to the openjdk lists here: [http://mail.openjdk.java.net/pipermail/nio-dev/2016-July/003784.html](http://mail.openjdk.java.net/pipermail/nio-dev/2016-July/003784.html) For now, maybe we can treat the overflowed results as unsigned long in our code, but IMO it does not really solve the problem... let's see what they say. ---
**Author:** ![gpiccinni](https://avatars.discourse-cdn.com/v4/letter/g/f19dbf/32.png) [@gpiccinni](https://discuss.elastic.co/u/gpiccinni)\ **Post date:** [August 4, 2016, 8:03am UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/11 "2016-08-04T08:03:58Z")
Hi I'm having the same issue. Using Oracle JDK doesn't solve the issue. ---
**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Post date:** [October 18, 2016, 2:14pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/12 "2016-10-18T14:14:19Z")
I have the same issue using EFS and Oracle JDK8u101 and ES v2.4.0 ---
**Author:** ![ivten](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@ivten](https://discuss.elastic.co/u/ivten)\ **Post date:** [October 18, 2016, 6:38pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/13 "2016-10-18T18:38:11Z")
The bug is now fixed in Elasticsearch 2.4.1 [https://github.com/elastic/elasticsearch/pull/20527](https://github.com/elastic/elasticsearch/pull/20527) I tested it and can confirm that it works. ---
**Author:** ![ZillaG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zillag/32/10505_2.png) [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Post date:** [October 27, 2016, 8:53pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/14 "2016-10-27T20:53:33Z")
+1 for ES v2.4.1 to fix this issue. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:08pm UTC](https://discuss.elastic.co/t/elasticsearch-with-amazon-elastic-file-system/55867/15 "2017-07-05T22:08:56Z")