ES - Yarn - HDFS or NFS


(Venkat Gunnu) #1

Hi

I am trying to run elastic search in yarn mode. Works fine if I have datapath set to local disk.

I am getting ClassNotFound exception if change the gateway.type to fs or hdfs. From documentation looks like only local gateway module is available and rest are deprecated.
https://www.elastic.co/guide/en/elasticsearch/reference/0.90/modules-gateway.html

I mounted the hadoop as NFS now I am getting Stale File Handle Exception. Any thoughts ?

2015-10-13 15:54:16,423][DEBUG][transport.netty ] [Shola Inkosi] connected to node [[Ani-Mator][Brt2Uuh1QSG5CHJESOPyyQ][n002][inet[/10.11.13.202:9300]]]
va:77)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.nio.file.FileSystemException: /mapr/apps/elasticsearch/distdata/data/elasticsearch/nodes/0/es.tmp: Stale file handle
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)


(Venkat Gunnu) #2

Ok Changing the datapath to
/apps/elasticsearch/${HOSTNAME}/data worked.
Now how to solve the restarts/node failures. In Yarn there is no guarantee same nodes are used after the restart. Is there any environment variable to get the nodeID like '0', '1', '2'. Any pointers are appreciated.


(system) #3