Unable to restore index from elasticsearch version 7.16.2 to 8.10.2

Elastic Stack Elasticsearch
1

I have elasticsearch with version 7.16.2 and index version is 7.6.2. Here GCS plugin is installed, GCS repo is configured and SLM is configured.

I have setup new cluster with with 8.10.2 version. Configured GCS repo. While trying to restore the index from GCS repo I am getting below error

[2024-02-21T08:06:34,260][INFO ][o.e.s.RestoreService     ] [es1] started restore of snapshot [myindex_gcsrepo:myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A] for indices [myindex]
[2024-02-21T08:06:34,561][WARN ][o.e.c.r.a.AllocationService] [es1] failing shard [FailedShard[routingEntry=[myindex][0], node[8D6AtnU4SRyD8EPzZw-ngQ], [P], recovery_source[snapshot recovery [ZI5M3XqWT3Odwj7Wcd1Ulw] from myindex_gcsrepo:myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A], s[INITIALIZING], a[id=GAQkSZVVQ1KT7rgS-A5H6w], unassigned_info[[reason=NEW_INDEX_RESTORED], at[2024-02-21T08:06:34.242Z], delayed=false, details[restore_source[myindex_gcsrepo/myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua]], allocation_status[no_attempt]], failed_attempts[0], expected_shard_size[2537419], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, markAsStale=true]]
org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:400) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$12(StoreRecovery.java:538) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$76(BlobStoreRepository.java:2955) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.10.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 20 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 17 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A]
	... 14 more
Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:620) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:280) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:385) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:407) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:117) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:34) ~[?:?]
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:235) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:172) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:125) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:214) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:65) ~[?:?]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3331) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:2983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
	... 3 more
[2024-02-21T08:06:34,722][WARN ][o.e.c.r.a.AllocationService] [es1] failing shard [FailedShard[routingEntry=[myindex][0], node[8D6AtnU4SRyD8EPzZw-ngQ], [P], recovery_source[snapshot recovery [ZI5M3XqWT3Odwj7Wcd1Ulw] from myindex_gcsrepo:myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A], s[INITIALIZING], a[id=kQ4BAcPnQ3iT2fiOpvzm0A], unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-21T08:06:34.561Z], failed_attempts[1], failed_nodes[[8D6AtnU4SRyD8EPzZw-ngQ]], delayed=false, last_node[8D6AtnU4SRyD8EPzZw-ngQ], details[failed shard on node [8D6AtnU4SRyD8EPzZw-ngQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:400)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$12(StoreRecovery.java:538)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$76(BlobStoreRepository.java:2955)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308)
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1623)
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 20 more
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 17 more
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A]
	... 14 more
Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
	at java.security.AccessController.checkPermission(AccessController.java:1071)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:620)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:280)
	at sun.net.www.http.HttpClient.New(HttpClient.java:385)
	at sun.net.www.http.HttpClient.New(HttpClient.java:407)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503)
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472)
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379)
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356)
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302)
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:117)
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:34)
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:112)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:235)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:172)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:112)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:125)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:214)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:65)
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3331)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:2983)
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	... 3 more
], allocation_status[fetching_shard_data]], failed_attempts[0], expected_shard_size[2537419], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, markAsStale=true]]
org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:400) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$12(StoreRecovery.java:538) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$76(BlobStoreRepository.java:2955) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.10.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 20 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 17 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A]
	... 14 more
Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:620) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:280) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:385) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:407) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:117) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:34) ~[?:?]
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:235) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:172) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:125) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:214) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:65) ~[?:?]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3331) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:2983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
	... 3 more
[2024-02-21T08:06:34,888][WARN ][o.e.c.r.a.AllocationService] [es1] failing shard [FailedShard[routingEntry=[myindex][0], node[8D6AtnU4SRyD8EPzZw-ngQ], [P], recovery_source[snapshot recovery [ZI5M3XqWT3Odwj7Wcd1Ulw] from myindex_gcsrepo:myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A], s[INITIALIZING], a[id=gM8K-5tdQhu0hWtV1ZL0ww], unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-21T08:06:34.721Z], failed_attempts[2], failed_nodes[[8D6AtnU4SRyD8EPzZw-ngQ]], delayed=false, last_node[8D6AtnU4SRyD8EPzZw-ngQ], details[failed shard on node [8D6AtnU4SRyD8EPzZw-ngQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:400)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$12(StoreRecovery.java:538)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$76(BlobStoreRepository.java:2955)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308)
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1623)
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 20 more
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 17 more
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A]
	... 14 more
Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
	at java.security.AccessController.checkPermission(AccessController.java:1071)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:620)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:280)
	at sun.net.www.http.HttpClient.New(HttpClient.java:385)
	at sun.net.www.http.HttpClient.New(HttpClient.java:407)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503)
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472)
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379)
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356)
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302)
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:117)
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:34)
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:112)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:235)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:172)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:112)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:125)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:214)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:65)
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3331)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:2983)
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	... 3 more
], allocation_status[fetching_shard_data]], failed_attempts[0], expected_shard_size[2537419], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, markAsStale=true]]
org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:400) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$12(StoreRecovery.java:538) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$76(BlobStoreRepository.java:2955) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.10.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 20 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 17 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A]
	... 14 more
Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:620) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:280) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:385) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:407) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:117) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:34) ~[?:?]
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:235) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:172) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:125) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:214) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:65) ~[?:?]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3331) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:2983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
	... 3 more
[2024-02-21T08:06:35,049][WARN ][o.e.c.r.a.AllocationService] [es1] failing shard [FailedShard[routingEntry=[myindex][0], node[8D6AtnU4SRyD8EPzZw-ngQ], [P], recovery_source[snapshot recovery [ZI5M3XqWT3Odwj7Wcd1Ulw] from myindex_gcsrepo:myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A], s[INITIALIZING], a[id=tzKOBXGKSQiWs_yyIOJ5dA], unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-21T08:06:34.888Z], failed_attempts[3], failed_nodes[[8D6AtnU4SRyD8EPzZw-ngQ]], delayed=false, last_node[8D6AtnU4SRyD8EPzZw-ngQ], details[failed shard on node [8D6AtnU4SRyD8EPzZw-ngQ]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:400)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$12(StoreRecovery.java:538)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$76(BlobStoreRepository.java:2955)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191)
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308)
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1623)
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 20 more
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 17 more
Caused by: [myindex/-M3U3j-xQfe6AweLQFRgSA][[myindex][0]] org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A]
	... 14 more
Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
	at java.security.AccessController.checkPermission(AccessController.java:1071)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:620)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:280)
	at sun.net.www.http.HttpClient.New(HttpClient.java:385)
	at sun.net.www.http.HttpClient.New(HttpClient.java:407)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503)
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472)
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379)
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356)
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302)
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:117)
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:34)
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:112)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:235)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:172)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:112)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:125)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:214)
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:65)
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3331)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:2983)
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	... 3 more
], allocation_status[fetching_shard_data]], failed_attempts[0], expected_shard_size[2537419], message=failed recovery, failure=org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, markAsStale=true]]
org.elasticsearch.indices.recovery.RecoveryFailedException: [myindex][0]: Recovery failed on {es2}{8D6AtnU4SRyD8EPzZw-ngQ}{WLcag8-QTeG5ECku7tNtJg}{es2}{192.168.1.124}{192.168.1.124:9300}{dm}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:400) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$12(StoreRecovery.java:538) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$76(BlobStoreRepository.java:2955) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.10.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 20 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 17 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [myindex-2024.02.21-i7acwhhjtmumdrljvnt2ua/MsgWm86uRhS1KCcXj1v55A]
	... 14 more
Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:905) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:620) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:280) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:385) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:407) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
	at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
	at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:117) ~[?:?]
	at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:34) ~[?:?]
	at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:235) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:172) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:112) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:125) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:214) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:65) ~[?:?]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:120) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3331) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$78(BlobStoreRepository.java:2983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
	... 3 more
2

Welcome!

Are you running all that locally?

Caused by: java.lang.SecurityException: access denied (""java.net.SocketPermission"" ""metadata.google.internal:80"" ""connect,resolve"")

This seems to indicate a connection issue between your Elasticsearch cluster and the Google API.

3

Hi

No, I am running on sever

I am able to connect

# # nc -vz metadata.google.internal 80
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 104.18.177.148:80.
Ncat: 0 bytes sent, 0 bytes received in 0.32 seconds.

OS Version: Alma Linux 9.3

4

As per compatibility matrix, my indices version 7.6.2 are compatible with 8.12.2 Elasticsearch version.

I have upgraded my Elasticsearch version to 8.12.2 and tried restoring the indices which were in 7.6.2 version. It worked.

I am not sure if it has to do anything with 8.10.2 version.

5

May be something has been fixed since then.

6

I have just upgraded Elasticsearch and restored indices

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.