# Elastic on Secure YARN

**URL:** <https://discuss.elastic.co/t/elastic-on-secure-yarn/23349>\
**Category:** Elasticsearch\
**Created:** [April 21, 2015, 7:47pm UTC](https://discuss.elastic.co/t/elastic-on-secure-yarn/23349 "2015-04-21T19:47:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eric\_Walk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_walk/32/748_2.png) [@Eric\_Walk](https://discuss.elastic.co/u/Eric_Walk)\
**Post date:** [April 21, 2015, 7:47pm UTC](https://discuss.elastic.co/t/elastic-on-secure-yarn/23349/1 "2015-04-21T19:47:35Z")

</div>

Hi All,

I'm working on running elasticsearch on yarn in a secured hadoop cluster  
(HDP 2.2).

It is failing when instantiating the containers with the error below. The  
issue appears that the Kerberos based JAAS Principal is not being passed  
from the launching container to the child containers. This could be a  
configuration issue, it could also just be something that is not  
implemented in the elasticsearch on yarn module. Was hoping to get a quick  
clarification before I spend too much time troubleshooting.

The files are uploaded in HDFS:

[root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls /apps/elasticsearch  
Found 2 items  
-rw-r--r-- 2 ewalk hdfs 31442427 2015-04-21 18:35  
/apps/elasticsearch/elasticsearch-1.5.1.zip  
-rw-r--r-- 2 ewalk hdfs 53819 2015-04-21 18:35  
/apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar

We are able to run other YARN jobs successfully, this is something specific  
to elastic's spin up process.

Any help would be appreciated.

Best,  
Eric

15/04/21 19:17:32 INFO am.ApplicationMaster: Starting ApplicationMaster...  
15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030  
15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0  
15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster with 2 nodes  
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0002.cedargatepartners.pvc:45454  
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0001.cedargatepartners.pvc:45454  
15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.  
15/04/21 19:17:38 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]  
15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...  
15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.  
Exception in thread "main" java.lang.IllegalArgumentException: Cannot find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip]; make sure the artifacts have been properly provisioned and the correct permissions are in place.  
at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)  
at org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)  
at org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)  
at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)  
at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)  
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ldevawshdp0002/10.4.0.7"; destination host is: "ldevawshdp0001.cedargatepartners.pvc":8020;  
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)  
at org.apache.hadoop.ipc.Client.call(Client.java:1472)  
at org.apache.hadoop.ipc.Client.call(Client.java:1399)  
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)  
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)  
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:606)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)  
at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)  
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)  
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)  
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)  
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)  
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)  
at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)  
... 4 more  
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]  
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:415)  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)  
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)  
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)  
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)  
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)  
at org.apache.hadoop.ipc.Client.call(Client.java:1438)  
... 21 more  
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]  
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)  
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)  
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)  
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)  
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)  
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:415)  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)  
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)  
... 24 more  
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)  
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)  
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)  
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)  
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)  
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)  
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)  
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)  
... 33 more

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [April 22, 2015, 10:10am UTC](https://discuss.elastic.co/t/elastic-on-secure-yarn/23349/2 "2015-04-22T10:10:47Z")

</div>

Hi,

Can you please raise an issue? From the looks of it the security token is indeed not passed along however I'm unclear  
whether you are configuring one yourself or not.  
You mention that launching other containers doesn't expose this issue, do you use them in a different way? Who holds the  
Kerberos credentials?

Cheers,

On 4/21/15 10:47 PM, Eric Walk wrote:

> Hi All,
> 
> I'm working on running elasticsearch on yarn in a secured hadoop cluster (HDP 2.2).
> 
> It is failing when instantiating the containers with the error below. The issue appears that the Kerberos based JAAS  
> Principal is not being passed from the launching container to the child containers. This could be a configuration issue,  
> it could also just be something that is not implemented in the elasticsearch on yarn module. Was hoping to get a quick  
> clarification before I spend too much time troubleshooting.
> 
> The files are uploaded in HDFS:
> 
> [root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls /apps/elasticsearch  
> Found 2 items  
> -rw-r--r-- 2 ewalk hdfs 31442427 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-1.5.1.zip  
> -rw-r--r-- 2 ewalk hdfs 53819 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar
> 
> We are able to run other YARN jobs successfully, this is something specific to elastic's spin up process.
> 
> Any help would be appreciated.
> 
> Best,  
> Eric
> 
> 15/04/21 19:17:32 INFO am.ApplicationMaster: Starting ApplicationMaster...  
> 15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
> 15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030  
> 15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0  
> 15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster with 2 nodes  
> 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0002.cedargatepartners.pvc:45454  
> 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0001.cedargatepartners.pvc:45454  
> 15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.  
> 15/04/21 19:17:38 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]  
> 15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...  
> 15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.  
> Exception in thread "main" java.lang.IllegalArgumentException: Cannot find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip]; make sure the artifacts have been properly provisioned and the correct permissions are in place.  
> at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)  
> at org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)  
> at org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)  
> at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)  
> at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)  
> Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ldevawshdp0002/10.4.0.7"; destination host is: "ldevawshdp0001.cedargatepartners.pvc":8020;  
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)  
> at org.apache.hadoop.ipc.Client.call(Client.java:1472)  
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)  
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)  
> at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)  
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:606)  
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)  
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)  
> at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)  
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)  
> at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)  
> at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)  
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)  
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)  
> at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)  
> ... 4 more  
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]  
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)  
> at java.security.AccessController.doPrivileged(Native Method)  
> at javax.security.auth.Subject.doAs(Subject.java:415)  
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)  
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)  
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)  
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)  
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)  
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)  
> ... 21 more  
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]  
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)  
> at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)  
> at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)  
> at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)  
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)  
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)  
> at java.security.AccessController.doPrivileged(Native Method)  
> at javax.security.auth.Subject.doAs(Subject.java:415)  
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)  
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)  
> ... 24 more  
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)  
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)  
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)  
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)  
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)  
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)  
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)  
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)  
> ... 33 more
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Eric\_Walk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_walk/32/748_2.png) [@Eric\_Walk](https://discuss.elastic.co/u/Eric_Walk)\
**Post date:** [April 22, 2015, 2:43pm UTC](https://discuss.elastic.co/t/elastic-on-secure-yarn/23349/3 "2015-04-22T14:43:10Z")

</div>

Hi Costin,

I'll go ahead and open an issue in Git. Here's a little more thought on the  
matter:

I'm not a developer, I'm an Infrastructure guy, so I don't know all the  
details of how it is generally done.

The issue occurs only when the containers that are launched to be the  
elastic cluster members try to access HDFS.

There are thousands of different applications that can run in YARN  
containers, many of them do not have this issue. For instance the Spark  
Hadoop RDD is able to successfully connect using this code:

> <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala>

> <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala>

Essentially, when you start an application in YARN, the master has the  
credentials of the user that launched it by default, but the application  
containers out on the node managers will not have any credentials unless  
they are specifically delegated out to them by the code running on the  
master.

Best,  
Eric

On Wed, Apr 22, 2015 at 6:11 AM Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote:

> Hi,
> 
> Can you please raise an issue? From the looks of it the security token is  
> indeed not passed along however I'm unclear  
> whether you are configuring one yourself or not.  
> You mention that launching other containers doesn't expose this issue, do  
> you use them in a different way? Who holds the  
> Kerberos credentials?
> 
> Cheers,
> 
> On 4/21/15 10:47 PM, Eric Walk wrote:
> 
> > Hi All,
> > 
> > I'm working on running elasticsearch on yarn in a secured hadoop cluster  
> > (HDP 2.2).
> > 
> > It is failing when instantiating the containers with the error below.  
> > The issue appears that the Kerberos based JAAS  
> > Principal is not being passed from the launching container to the child  
> > containers. This could be a configuration issue,  
> > it could also just be something that is not implemented in the  
> > elasticsearch on yarn module. Was hoping to get a quick  
> > clarification before I spend too much time troubleshooting.
> > 
> > The files are uploaded in HDFS:
> > 
> > [root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls  
> > /apps/elasticsearch  
> > Found 2 items  
> > -rw-r--r-- 2 ewalk hdfs 31442427 2015-04-21 18:35  
> > /apps/elasticsearch/elasticsearch-1.5.1.zip  
> > -rw-r--r-- 2 ewalk hdfs 53819 2015-04-21 18:35  
> > /apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar
> > 
> > We are able to run other YARN jobs successfully, this is something  
> > specific to elastic's spin up process.
> > 
> > Any help would be appreciated.
> > 
> > Best,  
> > Eric
> > 
> > 15/04/21 19:17:32 INFO am.ApplicationMaster: Starting  
> > ApplicationMaster...  
> > 15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load  
> > native-hadoop library for your platform... using builtin-java classes where  
> > applicable  
> > 15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at  
> > ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030  
> > 15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy:  
> > yarn.client.max-cached-nodemanagers-proxies : 0  
> > 15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster  
> > with 2 nodes  
> > 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for :  
> > ldevawshdp0002.cedargatepartners.pvc:45454  
> > 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for :  
> > ldevawshdp0001.cedargatepartners.pvc:45454  
> > 15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The  
> > short-circuit local reads feature cannot be used because libhadoop cannot  
> > be loaded.  
> > 15/04/21 19:17:38 WARN ipc.Client: Exception encountered while  
> > connecting to the server : javax.security.sasl.SaslException: GSS initiate  
> > failed [Caused by GSSException: No valid credentials provided (Mechanism  
> > level: Failed to find any Kerberos tgt)]  
> > 15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...  
> > 15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to  
> > be successfully unregistered.  
> > Exception in thread "main" java.lang.IllegalArgumentException: Cannot  
> > find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip];  
> > make sure the artifacts have been properly provisioned and the correct  
> > permissions are in place.  
> > at  
> > org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)  
> > at  
> > org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)  
> > at  
> > org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)  
> > at  
> > org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)  
> > at  
> > org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)  
> > Caused by: java.io.IOException: Failed on local exception:  
> > java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed  
> > [Caused by GSSException: No valid credentials provided (Mechanism level:  
> > Failed to find any Kerberos tgt)]; Host Details : local host is:  
> > "ldevawshdp0002/10.4.0.7"; destination host is:  
> > "ldevawshdp0001.cedargatepartners.pvc":8020;  
> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)  
> > at org.apache.hadoop.ipc.Client.call(Client.java:1472)  
> > at org.apache.hadoop.ipc.Client.call(Client.java:1399)  
> > at  
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)  
> > at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)  
> > at  
> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)  
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> > at  
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)  
> > at  
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> > at java.lang.reflect.Method.invoke(Method.java:606)  
> > at  
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)  
> > at  
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)  
> > at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)  
> > at  
> > org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)  
> > at  
> > org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)  
> > at  
> > org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)  
> > at  
> > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)  
> > at  
> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)  
> > at  
> > org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)  
> > ... 4 more  
> > Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS  
> > initiate failed [Caused by GSSException: No valid credentials provided  
> > (Mechanism level: Failed to find any Kerberos tgt)]  
> > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)  
> > at java.security.AccessController.doPrivileged(Native Method)  
> > at javax.security.auth.Subject.doAs(Subject.java:415)  
> > at  
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)  
> > at  
> > org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)  
> > at  
> > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)  
> > at  
> > org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)  
> > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)  
> > at org.apache.hadoop.ipc.Client.call(Client.java:1438)  
> > ... 21 more  
> > Caused by: javax.security.sasl.SaslException: GSS initiate failed  
> > [Caused by GSSException: No valid credentials provided (Mechanism level:  
> > Failed to find any Kerberos tgt)]  
> > at  
> > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)  
> > at  
> > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)  
> > at  
> > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)  
> > at  
> > org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)  
> > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)  
> > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)  
> > at java.security.AccessController.doPrivileged(Native Method)  
> > at javax.security.auth.Subject.doAs(Subject.java:415)  
> > at  
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)  
> > at  
> > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)  
> > ... 24 more  
> > Caused by: GSSException: No valid credentials provided (Mechanism level:  
> > Failed to find any Kerberos tgt)  
> > at  
> > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)  
> > at  
> > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)  
> > at  
> > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)  
> > at  
> > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)  
> > at  
> > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)  
> > at  
> > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)  
> > at  
> > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)  
> > ... 33 more
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)\>.  
> > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com)
> 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/6BGX8bKNWak/unsubscribe](https://groups.google.com/d/topic/elasticsearch/6BGX8bKNWak/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPSACbvQHAmJjmN3YxFC9c2sAYdean-BRai6HsiEFegOLEueDQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPSACbvQHAmJjmN3YxFC9c2sAYdean-BRai6HsiEFegOLEueDQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:18am UTC](https://discuss.elastic.co/t/elastic-on-secure-yarn/23349/4 "2017-07-06T00:18:17Z")

</div>