Hi @vee
I mean, i was able to run it fine without those permissions. I then added the permissions and it still worked as before, so the permissions did not alter my set-up in anyway.
/etc/java-11-openjdk/security/java.policy
I am running Debian Buster 10
Linux 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux
Could you please share the java.policy you used? May be I will give that a try. I see a default.policy at /jdk/lib/security/ but not sure if it covers all the required permissions.
Here... but i feel your issue might be with the setup of your private cloud
/
// This system policy file grants a set of default permissions to all domains
// and can be configured to grant additional permissions to modules and other
// code sources. The code source URL scheme for modules linked into a
// run-time image is "jrt".
//
// For example, to grant permission to read the "foo" property to the module
// "com.greetings", the grant entry is:
//
// grant codeBase "jrt:/com.greetings" {
// permission java.util.PropertyPermission "foo", "read";
// };
//
// default permissions granted to all domains
grant {
// allows anyone to listen on dynamic ports
permission java.net.SocketPermission "localhost:0", "listen";
// "standard" properies that can be read by anyone
permission java.util.PropertyPermission "java.version", "read";
permission java.util.PropertyPermission "java.vendor", "read";
permission java.util.PropertyPermission "java.vendor.url", "read";
permission java.util.PropertyPermission "java.class.version", "read";
permission java.util.PropertyPermission "os.name", "read";
permission java.util.PropertyPermission "os.version", "read";
permission java.util.PropertyPermission "os.arch", "read";
permission java.util.PropertyPermission "file.separator", "read";
permission java.util.PropertyPermission "path.separator", "read";
permission java.util.PropertyPermission "line.separator", "read";
permission java.util.PropertyPermission
"java.specification.version", "read";
permission java.util.PropertyPermission "java.specification.vendor", "read";
permission java.util.PropertyPermission "java.specification.name", "read";
permission java.util.PropertyPermission
"java.vm.specification.version", "read";
permission java.util.PropertyPermission
"java.vm.specification.vendor", "read";
permission java.util.PropertyPermission
"java.vm.specification.name", "read";
permission java.util.PropertyPermission "java.vm.version", "read";
permission java.util.PropertyPermission "java.vm.vendor", "read";
permission java.util.PropertyPermission "java.vm.name", "read";
//
permission java.lang.RuntimePermission accessDeclaredMembers
permission java.lang.RuntimePermission getClassLoader
permission java.lang.reflect.ReflectPermission suppressAccessChecks
permission java.net.SocketPermission * connect,resolve
permission java.util.PropertyPermission es.allow_insecure_settings read,write
};
@vee How did you install tar.gz deb? rpm?
There is something not right with the environment or something ...
Any chance you can just pull down an Elasticsearch 7.13.2 tar.gz , untar it , change no settings , start it and try to apply the same repository command... and see what we get?
PUT _snapshot/my_s3_repository
{
"type": "s3",
"settings": {
"bucket": "my-bucket",
"endpoint" : "s3host",
"protocol" : "http"
}
}
curl -X PUT "localhost:9200/_snapshot/my_s3_repository?pretty" -H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": {
"bucket": "my-bucket",
"endpoint" : "s3host",
"protocol" : "http"
}
}
'
@stephenb : We installed it by downloading the artifact and unzipping it (tar). Not through a package manager. We use ansible to rollout to all the environments, and so the same has been working across all environments. with no other issues so far. Although let me try from a different node and see if I can do a fresh installation.
@zx8086 : Tried to use same values in java.policy as part of the default.policy file but no luck 
//
// Permissions required by modules stored in a run-time image and loaded
// by the platform class loader.
//
// NOTE that this file is not intended to be modified. If additional
// permissions need to be granted to the modules in this file, it is
// recommended that they be configured in a separate policy file or
// ${java.home}/conf/security/java.policy.
//
grant codeBase "jrt:/java.compiler" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/java.net.http" {
permission java.lang.RuntimePermission "accessClassInPackage.sun.net";
permission java.lang.RuntimePermission "accessClassInPackage.sun.net.util";
permission java.lang.RuntimePermission "accessClassInPackage.sun.net.www";
permission java.lang.RuntimePermission "accessClassInPackage.jdk.internal.misc";
permission java.net.SocketPermission "*","connect,resolve";
permission java.net.URLPermission "http:*","*:*";
permission java.net.URLPermission "https:*","*:*";
permission java.net.URLPermission "ws:*","*:*";
permission java.net.URLPermission "wss:*","*:*";
permission java.net.URLPermission "socket:*","CONNECT"; // proxy
// For request/response body processors, fromFile, asFile
permission java.io.FilePermission "<<ALL FILES>>","read,write,delete";
permission java.util.PropertyPermission "*","read";
permission java.net.NetPermission "getProxySelector";
};
grant codeBase "jrt:/java.scripting" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/java.security.jgss" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/java.smartcardio" {
permission javax.smartcardio.CardPermission "*", "*";
permission java.lang.RuntimePermission "loadLibrary.j2pcsc";
permission java.lang.RuntimePermission
"accessClassInPackage.sun.security.jca";
permission java.lang.RuntimePermission
"accessClassInPackage.sun.security.util";
permission java.util.PropertyPermission
"javax.smartcardio.TerminalFactory.DefaultType", "read";
permission java.util.PropertyPermission "os.name", "read";
permission java.util.PropertyPermission "os.arch", "read";
permission java.util.PropertyPermission "sun.arch.data.model", "read";
permission java.util.PropertyPermission
"sun.security.smartcardio.library", "read";
permission java.util.PropertyPermission
"sun.security.smartcardio.t0GetResponse", "read";
permission java.util.PropertyPermission
"sun.security.smartcardio.t1GetResponse", "read";
permission java.util.PropertyPermission
"sun.security.smartcardio.t1StripLe", "read";
// needed for looking up native PC/SC library
permission java.io.FilePermission "<<ALL FILES>>","read";
permission java.security.SecurityPermission "putProviderProperty.SunPCSC";
permission java.security.SecurityPermission
"clearProviderProperties.SunPCSC";
permission java.security.SecurityPermission
"removeProviderProperty.SunPCSC";
};
grant codeBase "jrt:/java.sql" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/java.sql.rowset" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/java.xml.crypto" {
permission java.lang.RuntimePermission
"getStackWalkerWithClassReference";
permission java.lang.RuntimePermission
"accessClassInPackage.sun.security.util";
permission java.util.PropertyPermission "*", "read";
permission java.security.SecurityPermission "putProviderProperty.XMLDSig";
permission java.security.SecurityPermission
"clearProviderProperties.XMLDSig";
permission java.security.SecurityPermission
"removeProviderProperty.XMLDSig";
permission java.security.SecurityPermission
"com.sun.org.apache.xml.internal.security.register";
permission java.security.SecurityPermission
"getProperty.jdk.xml.dsig.secureValidationPolicy";
permission java.lang.RuntimePermission
"accessClassInPackage.com.sun.org.apache.xml.internal.*";
permission java.lang.RuntimePermission
"accessClassInPackage.com.sun.org.apache.xpath.internal";
permission java.lang.RuntimePermission
"accessClassInPackage.com.sun.org.apache.xpath.internal.*";
};
grant codeBase "jrt:/jdk.accessibility" {
permission java.lang.RuntimePermission "accessClassInPackage.sun.awt";
};
grant codeBase "jrt:/jdk.charsets" {
permission java.util.PropertyPermission "os.name", "read";
permission java.lang.RuntimePermission "charsetProvider";
permission java.lang.RuntimePermission
"accessClassInPackage.jdk.internal.misc";
permission java.lang.RuntimePermission "accessClassInPackage.sun.nio.cs";
};
grant codeBase "jrt:/jdk.crypto.ec" {
permission java.lang.RuntimePermission
"accessClassInPackage.sun.security.*";
permission java.lang.RuntimePermission "loadLibrary.sunec";
permission java.security.SecurityPermission "putProviderProperty.SunEC";
permission java.security.SecurityPermission "clearProviderProperties.SunEC";
permission java.security.SecurityPermission "removeProviderProperty.SunEC";
};
grant codeBase "jrt:/jdk.crypto.cryptoki" {
permission java.lang.RuntimePermission
"accessClassInPackage.sun.security.*";
permission java.lang.RuntimePermission "accessClassInPackage.sun.nio.ch";
permission java.lang.RuntimePermission "loadLibrary.j2pkcs11";
permission java.util.PropertyPermission "sun.security.pkcs11.allowSingleThreadedModules", "read";
permission java.util.PropertyPermission "sun.security.pkcs11.disableKeyExtraction", "read";
permission java.util.PropertyPermission "os.name", "read";
permission java.util.PropertyPermission "os.arch", "read";
permission java.util.PropertyPermission "jdk.crypto.KeyAgreement.legacyKDF", "read";
permission java.security.SecurityPermission "putProviderProperty.*";
permission java.security.SecurityPermission "clearProviderProperties.*";
permission java.security.SecurityPermission "removeProviderProperty.*";
permission java.security.SecurityPermission
"getProperty.auth.login.defaultCallbackHandler";
permission java.security.SecurityPermission "authProvider.*";
// Needed for reading PKCS11 config file and NSS library check
permission java.io.FilePermission "<<ALL FILES>>", "read";
};
grant codeBase "jrt:/jdk.dynalink" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.httpserver" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.internal.le" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.internal.vm.compiler" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.internal.vm.compiler.management" {
permission java.lang.RuntimePermission "accessClassInPackage.jdk.internal.vm.compiler.collections";
permission java.lang.RuntimePermission "accessClassInPackage.jdk.vm.ci.runtime";
permission java.lang.RuntimePermission "accessClassInPackage.jdk.vm.ci.services";
permission java.lang.RuntimePermission "accessClassInPackage.org.graalvm.compiler.core.common";
permission java.lang.RuntimePermission "accessClassInPackage.org.graalvm.compiler.debug";
permission java.lang.RuntimePermission "accessClassInPackage.org.graalvm.compiler.hotspot";
permission java.lang.RuntimePermission "accessClassInPackage.org.graalvm.compiler.options";
permission java.lang.RuntimePermission "accessClassInPackage.org.graalvm.compiler.phases.common.jmx";
permission java.lang.RuntimePermission "accessClassInPackage.org.graalvm.compiler.serviceprovider";
};
grant codeBase "jrt:/jdk.jsobject" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.localedata" {
permission java.lang.RuntimePermission "accessClassInPackage.sun.text.*";
permission java.lang.RuntimePermission "accessClassInPackage.sun.util.*";
};
grant codeBase "jrt:/jdk.naming.dns" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.scripting.nashorn" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.scripting.nashorn.shell" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.security.auth" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.security.jgss" {
permission java.security.AllPermission;
};
grant codeBase "jrt:/jdk.zipfs" {
permission java.io.FilePermission "<<ALL FILES>>", "read,write,delete";
permission java.lang.RuntimePermission "fileSystemProvider";
permission java.lang.RuntimePermission "accessUserInformation";
permission java.util.PropertyPermission "os.name", "read";
permission java.util.PropertyPermission "user.dir", "read";
permission java.util.PropertyPermission "user.name", "read";
};
// permissions needed by applications using java.desktop module
grant {
permission java.lang.RuntimePermission "accessClassInPackage.com.sun.beans";
permission java.lang.RuntimePermission "accessClassInPackage.com.sun.beans.*";
permission java.lang.RuntimePermission "accessClassInPackage.com.sun.java.swing.plaf.*";
permission java.lang.RuntimePermission "accessClassInPackage.com.apple.*";
};
grant {
// allows anyone to listen on dynamic ports
permission java.net.SocketPermission "localhost:0", "listen";
// "standard" properies that can be read by anyone
permission java.util.PropertyPermission "java.version", "read";
permission java.util.PropertyPermission "java.vendor", "read";
permission java.util.PropertyPermission "java.vendor.url", "read";
permission java.util.PropertyPermission "java.class.version", "read";
permission java.util.PropertyPermission "os.name", "read";
permission java.util.PropertyPermission "os.version", "read";
permission java.util.PropertyPermission "os.arch", "read";
permission java.util.PropertyPermission "file.separator", "read";
permission java.util.PropertyPermission "path.separator", "read";
permission java.util.PropertyPermission "line.separator", "read";
permission java.util.PropertyPermission
"java.specification.version", "read";
permission java.util.PropertyPermission "java.specification.vendor", "read";
permission java.util.PropertyPermission "java.specification.name", "read";
permission java.util.PropertyPermission
"java.vm.specification.version", "read";
permission java.util.PropertyPermission
"java.vm.specification.vendor", "read";
permission java.util.PropertyPermission
"java.vm.specification.name", "read";
permission java.util.PropertyPermission "java.vm.version", "read";
permission java.util.PropertyPermission "java.vm.vendor", "read";
permission java.util.PropertyPermission "java.vm.name", "read";
//
permission java.lang.RuntimePermission "accessDeclaredMembers";
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission java.net.SocketPermission "*", "connect,resolve";
permission java.util.PropertyPermission "es.allow_insecure_settings", "read,write";
};
If it is still pre-pending the bucket name to the URL none the rest matters I think....
Tried to install ES on a new VM, downloaded v7.13.2, untar and updated the Elasticsearch.yml and other required config files. I see some warning messages with bucket prefix on the hostname.
Elasticsearch.yml:
bootstrap.memory_lock: true
cluster.name: test-cluster
cluster.remote.connect: false
discovery.type: single-node
http.port: 9200
node.data: true
node.ingest: true
node.master: true
transport.port: 9300
node.name: ingest1
path.data: <path-to-data>/ingest1/data
path.logs: <path-to-logs>
network.host: hostname
network.publish_host: hostname
network.bind_host: hostname
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false
xpack.security.http.ssl.enabled: false
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.exclude: "access_granted"
Here are all the warnings that I see when I run the create snapshot command:
PUT _snapshot/my_s3_repository
{
"type": "s3",
"settings": {
"bucket": "bucket1",
"endpoint": "host:port",
"protocol": "http"
}
}
Enabled trace and here are the key line entries from stack trace:
101 [2021-10-15T13:01:16,493][WARN ][c.a.j.SdkMBeanRegistrySupport] [ingest1]
102 java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
.
.
138 [2021-10-15T13:01:16,500][WARN ][c.a.m.AwsSdkMetrics ] [ingest1]
139 java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
.
.
175 [2021-10-15T13:01:16,515][WARN ][c.a.s.s.i.UseArnRegionResolver] [ingest1] Unable to load config file null
176 java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/rabbitmq/.aws/config" "read")
.
.
215 [2021-10-15T13:01:16,606][DEBUG][c.a.m.CsmConfigurationProviderChain] [ingest1] Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProv ider@66a74051: Unable to load Client Side Monitoring configurations from environment variables!
216 [2021-10-15T13:01:16,606][DEBUG][c.a.m.CsmConfigurationProviderChain] [ingest1] Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@ 42e4e92a: Unable to load Client Side Monitoring configurations from system properties variables!
217 [2021-10-15T13:01:16,608][DEBUG][c.a.m.CsmConfigurationProviderChain] [ingest1] Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@60edb19 0: Unable to load config file
.
.
241 [2021-10-15T13:01:16,749][DEBUG][c.a.h.c.ClientConnectionManagerFactory] [ingest1]
242 java.lang.reflect.InvocationTargetException: null
.
.
302 [2021-10-15T13:01:16,754][DEBUG][c.a.request ] [ingest1] Retrying Request: PUT http://bucket1.host:port /tests-uxc-LcwNTbWVZbvm1CdNng/master.dat Headers: (amz-sdk-invocation-id: 4f63df5f-dd24-597a-3844-7bdab523e8e6, Content-Length: 22, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/4.18.0-305 .10.2.el8_4.x86_64 OpenJDK_64-Bit_Server_VM/16+36 java/16 vendor/AdoptOpenJDK, x-amz-acl: private, x-amz-storage-class: STANDARD, )
303 [2021-10-15T13:01:16,755][DEBUG][c.a.h.AmazonHttpClient ] [ingest1] Retriable error detected, will retry in 77ms, attempt number: 0
304 [2021-10-15T13:01:16,833][DEBUG][c.a.a.AWS4Signer ] [ingest1] AWS4 Canonical Request: '"PUT
.
.
325 [2021-10-15T13:01:16,835][DEBUG][c.a.h.c.ClientConnectionManagerFactory] [ingest1]
326 java.lang.reflect.InvocationTargetException: null
.
.
373 Caused by: java.net.UnknownHostException: bucket1.host
.
.
548 org.elasticsearch.repositories.RepositoryVerificationException: [my_s3_repository] path is not accessible on master node
549 Caused by: java.io.IOException: Unable to upload object [tests-uxc-LcwNTbWVZbvm1CdNng/master.dat] using a single upload
Paste bin: Full stack trace
https://pastebin.pl/view/cdbeee05
Thanks for your time!!
Perhaps try
"path_style_access" : true
PUT _snapshot/my_s3_repository
{
"type": "s3",
"settings": {
"bucket": "my-bucket",
"endpoint" : "hostname:5500",
"protocol" : "http",
"path_style_access" : true
}
}
https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html
Wohoo!! It finally worked on the stand alone node after adding the property you shared @stephenb . Thanks so much for that. I was able to connect using https endpoint as well just fine from the test node. I did try this property earlier but the value true was listed as string and not boolean and so it didn't work.
"path_style_access" : "true"
Darn ... should have seen that earlier!
But good to know... now we are all experts 
Absolutely! I'm glad it's resolved after much struggle. Truly appreciate the help I got from this forum here!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.