Elastic on Secure YARN

Eric_Walk · April 21, 2015, 7:47pm

Hi All,

I'm working on running elasticsearch on yarn in a secured hadoop cluster
(HDP 2.2).

It is failing when instantiating the containers with the error below. The
issue appears that the Kerberos based JAAS Principal is not being passed
from the launching container to the child containers. This could be a
configuration issue, it could also just be something that is not
implemented in the elasticsearch on yarn module. Was hoping to get a quick
clarification before I spend too much time troubleshooting.

The files are uploaded in HDFS:

[root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls /apps/elasticsearch
Found 2 items
-rw-r--r-- 2 ewalk hdfs 31442427 2015-04-21 18:35
/apps/elasticsearch/elasticsearch-1.5.1.zip
-rw-r--r-- 2 ewalk hdfs 53819 2015-04-21 18:35
/apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar

We are able to run other YARN jobs successfully, this is something specific
to elastic's spin up process.

Any help would be appreciated.

Best,
Eric

15/04/21 19:17:32 INFO am.ApplicationMaster: Starting ApplicationMaster...
15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030
15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster with 2 nodes
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0002.cedargatepartners.pvc:45454
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0001.cedargatepartners.pvc:45454
15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/04/21 19:17:38 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...
15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
Exception in thread "main" java.lang.IllegalArgumentException: Cannot find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip]; make sure the artifacts have been properly provisioned and the correct permissions are in place.
at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)
at org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)
at org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)
at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)
at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ldevawshdp0002/10.4.0.7"; destination host is: "ldevawshdp0001.cedargatepartners.pvc":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)
... 4 more
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 21 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 24 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 33 more

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

costin · April 22, 2015, 10:10am

Hi,

Can you please raise an issue? From the looks of it the security token is indeed not passed along however I'm unclear
whether you are configuring one yourself or not.
You mention that launching other containers doesn't expose this issue, do you use them in a different way? Who holds the
Kerberos credentials?

Cheers,

On 4/21/15 10:47 PM, Eric Walk wrote:

Hi All,

I'm working on running elasticsearch on yarn in a secured hadoop cluster (HDP 2.2).

It is failing when instantiating the containers with the error below. The issue appears that the Kerberos based JAAS
Principal is not being passed from the launching container to the child containers. This could be a configuration issue,
it could also just be something that is not implemented in the elasticsearch on yarn module. Was hoping to get a quick
clarification before I spend too much time troubleshooting.

The files are uploaded in HDFS:

[root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls /apps/elasticsearch
Found 2 items
-rw-r--r-- 2 ewalk hdfs 31442427 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-1.5.1.zip
-rw-r--r-- 2 ewalk hdfs 53819 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar

We are able to run other YARN jobs successfully, this is something specific to elastic's spin up process.

Any help would be appreciated.

Best,
Eric

15/04/21 19:17:32 INFO am.ApplicationMaster: Starting ApplicationMaster...
15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030
15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster with 2 nodes
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0002.cedargatepartners.pvc:45454
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0001.cedargatepartners.pvc:45454
15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/04/21 19:17:38 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...
15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
Exception in thread "main" java.lang.IllegalArgumentException: Cannot find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip]; make sure the artifacts have been properly provisioned and the correct permissions are in place.
at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)
at org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)
at org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)
at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)
at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ldevawshdp0002/10.4.0.7"; destination host is: "ldevawshdp0001.cedargatepartners.pvc":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)
... 4 more
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 21 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 24 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 33 more

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Eric_Walk · April 22, 2015, 2:43pm

Hi Costin,

I'll go ahead and open an issue in Git. Here's a little more thought on the
matter:

I'm not a developer, I'm an Infrastructure guy, so I don't know all the
details of how it is generally done.

The issue occurs only when the containers that are launched to be the
elastic cluster members try to access HDFS.

There are thousands of different applications that can run in YARN
containers, many of them do not have this issue. For instance the Spark
Hadoop RDD is able to successfully connect using this code:

github.com

apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.spark.rdd

import java.io.{FileNotFoundException, IOException}

This file has been truncated. show original

github.com

apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.spark.deploy

import java.io.{ByteArrayInputStream, ByteArrayOutputStream, DataInputStream, DataOutputStream, File, IOException}

This file has been truncated. show original

Essentially, when you start an application in YARN, the master has the
credentials of the user that launched it by default, but the application
containers out on the node managers will not have any credentials unless
they are specifically delegated out to them by the code running on the
master.

Best,
Eric

On Wed, Apr 22, 2015 at 6:11 AM Costin Leau costin.leau@gmail.com wrote:

Hi,

Can you please raise an issue? From the looks of it the security token is
indeed not passed along however I'm unclear
whether you are configuring one yourself or not.
You mention that launching other containers doesn't expose this issue, do
you use them in a different way? Who holds the
Kerberos credentials?

Cheers,

On 4/21/15 10:47 PM, Eric Walk wrote:

Hi All,

I'm working on running elasticsearch on yarn in a secured hadoop cluster
(HDP 2.2).

It is failing when instantiating the containers with the error below.
The issue appears that the Kerberos based JAAS
Principal is not being passed from the launching container to the child
containers. This could be a configuration issue,
it could also just be something that is not implemented in the
elasticsearch on yarn module. Was hoping to get a quick
clarification before I spend too much time troubleshooting.

The files are uploaded in HDFS:

[root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls
/apps/elasticsearch
Found 2 items
-rw-r--r-- 2 ewalk hdfs 31442427 2015-04-21 18:35
/apps/elasticsearch/elasticsearch-1.5.1.zip
-rw-r--r-- 2 ewalk hdfs 53819 2015-04-21 18:35
/apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar

We are able to run other YARN jobs successfully, this is something
specific to elastic's spin up process.

Any help would be appreciated.

Best,
Eric

15/04/21 19:17:32 INFO am.ApplicationMaster: Starting
ApplicationMaster...
15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at
ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030
15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster
with 2 nodes
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for :
ldevawshdp0002.cedargatepartners.pvc:45454
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for :
ldevawshdp0001.cedargatepartners.pvc:45454
15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The
short-circuit local reads feature cannot be used because libhadoop cannot
be loaded.
15/04/21 19:17:38 WARN ipc.Client: Exception encountered while
connecting to the server : javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]
15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...
15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to
be successfully unregistered.
Exception in thread "main" java.lang.IllegalArgumentException: Cannot
find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip];
make sure the artifacts have been properly provisioned and the correct
permissions are in place.
at
org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)
at
org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)
at
org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)
at
org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)
at
org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)
Caused by: java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]; Host Details : local host is:
"ldevawshdp0002/10.4.0.7"; destination host is:
"ldevawshdp0001.cedargatepartners.pvc":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at
org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)
... 4 more
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at
org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 21 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at
org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 24 more
Caused by: GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 33 more

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearch+unsubscribe@googlegroups.com <mailto:
elasticsearch+unsubscribe@googlegroups.com>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com

<
https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/6BGX8bKNWak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPSACbvQHAmJjmN3YxFC9c2sAYdean-BRai6HsiEFegOLEueDQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.