HDFS snapshot java access denied

[tarp (Tim Arp)]

Hi,

I'm trying to setup HDFS as a snapshot respository. I'm able to create the repo. But upon creating the first snapshot, I'm getting these errors.
"shards": {
"0": {
"stage": "FAILURE",
"stats": {
"number_of_files": 0,
"processed_files": 0,
"total_size_in_bytes": 0,
"processed_size_in_bytes": 0,
"start_time_in_millis": 0,
"time_in_millis": 0
},
"reason": """IndexShardSnapshotFailedException[Failed to snapshot]; nested: ElasticsearchException[failed to create blob container]; nested: IOException[com.google.protobuf.ServiceException: java.security.AccessControlException: access denied ("javax.security.auth.PrivateCredentialPermission" "org.apache.hadoop.security.Credentials" "read")]; nested: ServiceException[java.security.AccessControlException: access denied ("javax.security.auth.PrivateCredentialPermission" "org.apache.hadoop.security.Credentials" "read")]; nested: AccessControlException[access denied ("javax.security.auth.PrivateCredentialPermission" "org.apache.hadoop.security.Credentials" "read")]; """
}

I tried to add the JVM Option -Djava.security.policy=file:///usr/share/elasticsearch/plugins/repository-hdfs/plugin-security.policy
But that didn't help.
Any ideas?
Thanks,
Tim

[james.baiera]

Hi @tarp, which version of ES and the plugin are you using?

[tarp (Tim Arp)]

Hi James,
You are a miracle worker, I just tried it and it is working!
I hadn't worked on this in a while, previously I was on 5.5.2 for ES with the latest plugin I could download. Since then I have upgraded this cluster to 5.6.0 and I believe the puppet module grabs the latest plugin. Anyway, just tried it and it's backing up. So that's good. I will also have to test this on our prod cluster, but I need to talk to our HDFS team first to get the connection info.
thanks for the reply.
--Tim

[james.baiera]

@tarp Huh... I'm happy I could help? Let me know if the issue comes back at all