I have a Elasticsearch cluster which I have just migrated from 5.6.4 to 6.5.1. (This version was selected because it is the highest version currently supported by the elastic4s library used by our code.) Unfortunately, this has caused a problem with snapshotting; when I try it I get a java.lang.SecurityException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers"). (It all worked fine before the upgrade.)
We're running Elasticsearch in a Docker container (on Linux) based on elasticsearch:6.5.1, but I have also seen the problem when running Elasticsearch standalone (on MacOS); it also occurs whether running with multiple nodes (in our staging environment) or a single node (on my Macbook). It also occurs in later 6.x versions.
To reproduce the problem it is not necessary to do any more than run Elasticsearch with the repository-gcs plugin installed, then PUT { "type":"gcs", "settings": {"bucket":"elasticsearch-backup-store-[REDACTED]-staging"}} to /_snapshot/elasticsearch_backup_repository.
The problem occurs despite the grant { [...] permission java.lang.RuntimePermission "accessDeclaredMembers"; [...] }; in /usr/share/elasticsearch/plugins/repository-gcs/plugin-security.policy; it even occurs if I try to add the grant globally in /opt/jdk-11.0.1/conf/security/java.policy or /opt/jdk-11.0.1/lib/security/default.policy.
I tried running ES with -Djava.security.debug=access,failure, and observed a curious thing: Whilst other security grant denials were reported with information about the domain that caused the security breach, this one was not, which suggests to me that possibly repository-gcs is not running with the same security setup as the main Elasticsearch executable.
Here's the full stacktrace of the problem (edited down slightly to get under the character limit):
[INFO ][o.e.r.RepositoriesService] [kPIR_P5] put repository [elasticsearch_backup_repository]
[WARN ][o.e.r.g.GoogleCloudStorageService] [kPIR_P5] "Application Default Credentials" are not supported out of the box. Additional file system permissions have to be granted to the plugin.
[WARN ][r.suppressed ] [kPIR_P5] path: /_snapshot/elasticsearch_backup_repository, params: {repository=elasticsearch_backup_repository}
org.elasticsearch.repositories.RepositoryException: [elasticsearch_backup_repository] cannot create blob store
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:336) ~[elasticsearch-6.5.1.jar:6.5.1]
at o.e.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:635) ~[elasticsearch-6.5.1.jar:6.5.1]
at o.e.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:218) [elasticsearch-6.5.1.jar:6.5.1]
at o.e.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
Caused by: org.elasticsearch.common.blobstore.BlobStoreException: Unable to check if bucket [elasticsearch-backup-store-redimapl-staging] exists
at o.e.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:111)
at o.e.repositories.gcs.GoogleCloudStorageBlobStore.<init>(GoogleCloudStorageBlobStore.java:77)
at o.e.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:135)
at o.e.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:46)
at o.e.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:332) ~[elasticsearch-6.5.1.jar:6.5.1]
... 6 more
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:895)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
at java.lang.Class.checkMemberAccess(Class.java:2848)
at java.lang.Class.getDeclaredFields(Class.java:2247)
at com.google.api.client.util.ClassInfo.<init>(ClassInfo.java:171)
at c.g.a.c.util.ClassInfo.of(ClassInfo.java:90)
at c.g.a.c.util.ClassInfo.<init>(ClassInfo.java:193)
at c.g.a.c.util.ClassInfo.of(ClassInfo.java:90)
at c.g.a.c.util.ClassInfo.<init>(ClassInfo.java:193)
at c.g.a.c.util.ClassInfo.of(ClassInfo.java:90)
at c.g.a.c.util.GenericData.<init>(GenericData.java:79)
at c.g.a.c.util.GenericData.<init>(GenericData.java:61)
at c.g.a.c.http.GenericUrl.<init>(GenericUrl.java:157)
at c.g.a.c.http.GenericUrl.<init>(GenericUrl.java:142)
at c.g.a.c.http.GenericUrl.<init>(GenericUrl.java:114)
at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:452)
at c.g.c.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:431)
at c.g.c.ServiceOptions.getDefaultProjectId(ServiceOptions.java:339)
at c.g.c.ServiceOptions.getDefaultProject(ServiceOptions.java:317)
at c.g.c.ServiceOptions.<init>(ServiceOptions.java:270)
at c.g.c.storage.StorageOptions.<init>(StorageOptions.java:83)
at c.g.c.storage.StorageOptions.<init>(StorageOptions.java:31)
at c.g.c.storage.StorageOptions$Builder.build(StorageOptions.java:78)
at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:211)
at o.e.repositories.gcs.GoogleCloudStorageService.lambda$refreshAndClearCache$0(GoogleCloudStorageService.java:89)
at o.e.common.util.LazyInitializable.maybeCompute(LazyInitializable.java:103) ~[elasticsearch-6.5.1.jar:6.5.1]
at o.e.common.util.LazyInitializable.getOrCompute(LazyInitializable.java:81) ~[elasticsearch-6.5.1.jar:6.5.1]
at o.e.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:114)
at o.e.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:83)
at o.e.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:108)
at java.security.AccessController.doPrivileged(Native Method)
at o.e.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44)
at o.e.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:108)
at o.e.repositories.gcs.GoogleCloudStorageBlobStore.<init>(GoogleCloudStorageBlobStore.java:77)
at o.e.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:135)
at o.e.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:46)
at o.e.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:332) ~[elasticsearch-6.5.1.jar:6.5.1]
... 6 more
Any help would be appreciated.