Azure snapshot error: Unable to find client with name [abc]

pjd · August 1, 2024, 11:07pm

Running elasticsearch as a kubernetes pod. Here are the steps followed.

added keystore entries
./bin/elasticsearch-keystore add azure.client.default.account
./bin/elasticsearch-keystore add azure.client.default.key
Ran reload
curl -X POST localhost:9200/_nodes/reload_secure_settings
Fails to create repository with error
Failed creating repository for cluster: repositoryId: 2024-08-01-day, error: 500 Internal Server Error: "{"error":{"root_cause":[{"type":"settings_exception","reason":"Unable to find client with name [abc]"}],"type":"repository_verification_exception","reason":"[2024-08-01-day] path [][2024-08-01-day] is not accessible on master node","caused_by":{"type":"settings_exception","reason":"Unable to find client with name [abc]"}},"status":500}"]

dadoonet · August 2, 2024, 2:46pm

Welcome!

What is the command you ran to create the repository?

If you did something like:

PUT _snapshot/2024-08-01-day
{
  "type": "azure",
  "settings": {
    "client": "abc"
  }
}

Then this is wrong as the client id you defined is default.

Check Azure repository | Elasticsearch Guide [8.14] | Elastic

pjd · August 2, 2024, 9:46pm

I did not have the client set in the body. So I assumed it would take the "default"

could you please let me know what are the causes of this error ? thanks

dadoonet · August 2, 2024, 10:06pm

Could you share the API call please?
And what gives:

GET /_snapshot/_all

pjd · August 4, 2024, 8:28pm

Sure. Here is the output of _/snapshot/_all and sample create repository payload.

{
  "2024-08-02-day": {
    "type": "azure",
    "settings": {
      "container": "cluster",
      "base_path": "gold-0001/2024-08-02-day",
      "endpoint": "https://abc.blob.core.windows.net/",
      "account": "abc",
      "key": "somekey"
    }
  },
  "2024-08-03-day": {
    "type": "azure",
    "settings": {
      "container": "cluster",
      "base_path": "gold-0001/2024-08-03-day",
      "endpoint": "https://abc.blob.core.windows.net/",
      "account": "abc",
      "key": "somekey"
    }
  },
  "2024-08-04-day": {
    "type": "azure",
    "settings": {
      "container": "cluster",
      "base_path": "gold-0001/2024-08-04-day",
      "endpoint": "https://abc.blob.core.windows.net/",
      "account": "abc",
      "key": "somekey"
    }
  }
}

{type=azure, settings={container=cluster, account=abc, key=somekey, base_path=gold-0001/2024-08-03-day}}

And when I run list on elasticsearch-keystore on same node, I do see azure account and key.

bash-4.4$ ./elasticsearch-keystore list

azure.client.default.account

azure.client.default.key

keystore.seed

dadoonet · August 4, 2024, 9:21pm

From Elastic Search to Elasticsearch

dadoonet · August 5, 2024, 8:39am

What is the request you are sending to the cluster which creates this error message?

pjd · August 5, 2024, 2:39pm

Looks like it runs into this error when creating snaposhot. Here is the stacktrace.

[2024-08-05T14:30:00,741][WARN ][r.suppressed       ] [elasticsearch-gold-0001-data-0] path: /_snapshot/2024-08-05-day/20240805143000, params: {repository=2024-08-05-day, snapshot=20240805143000}
org.elasticsearch.transport.RemoteTransportException: [elasticsearch-gold-0001-master-0][10.42.0.18:9310][cluster:admin/snapshot/create]
Caused by: org.elasticsearch.repositories.RepositoryException: [2024-08-05-day] Could not determine repository generation from root blobs
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1895) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.16.jar:7.17.16]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Unable to list blobs by prefix [index-] for path gold-0001/2024-08-05-day/
	at org.elasticsearch.repositories.azure.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:340) ~[?:?]
	at org.elasticsearch.repositories.azure.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:135) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2597) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2569) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1892) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.16.jar:7.17.16]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.elasticsearch.common.settings.SettingsException: Unable to find client with name [abc]
	at org.elasticsearch.repositories.azure.AzureStorageService.getClientSettings(AzureStorageService.java:101) ~[?:?]
	at org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:91) ~[?:?]
	at org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:624) ~[?:?]
	at org.elasticsearch.repositories.azure.AzureBlobStore.client(AzureBlobStore.java:616) ~[?:?]
	at org.elasticsearch.repositories.azure.AzureBlobStore.listBlobsByPrefix(AzureBlobStore.java:321) ~[?:?]
	at org.elasticsearch.repositories.azure.AzureBlobContainer.listBlobsByPrefix(AzureBlobContainer.java:135) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2597) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2569) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1892) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.16.jar:7.17.16]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.16.jar:7.17.16]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

dadoonet · August 5, 2024, 5:20pm

So. My advices:

upgrade to latest 7.17
Add an abc client using the same credentials you used for default

You have something probably wrong in your cluster but this should help.

pjd · August 8, 2024, 3:08pm

Thank you for the suggestions.

We removed account, key and endpoint from create repository payload and that seem to have fixed the issue. I am not sure if these were added to make it work with older version of elasticsearch.

Another follow-up question for elastic 7.17. It looks like we need to add azure account name and key in elastic keystore in every node using the add commands. Is there a way to do this from helm charts in kubernetes so that the manual step of adding in keystore can be avoided ? Adding manually in every node when there are huge numbers of nodes would be difficult.

Thanks

dadoonet · August 8, 2024, 7:45pm

I have no idea. But I'd open a new discussion for this.

system · September 5, 2024, 7:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Snapshot error: Unable to find client with name [default] Elastic Cloud on Kubernetes (ECK) snapshot-and-restore	3	2804	March 25, 2021
Experiencing an error while restoring the snapshot Elastic Search elastic-app-search	7	285	March 19, 2024
Unable to create snapshot repo in azure Elasticsearch snapshot-and-restore	6	940	November 15, 2022
Azure snapshot missing required setting: azure.client.default.account Elasticsearch	2	1354	October 18, 2019
ElasticSearch 7.3 Azure repository snapshot error Elasticsearch docker	4	1784	October 22, 2019

Azure snapshot error: Unable to find client with name [abc]

Related topics