Unable to create snapshot repo in azure

Im trying to register an azure blob as snapshot repository. I have already tested this with elasticsearch 7.17 on test VMs outside my firm but it is failing when Im trying the same steps inside the firm.

My first impression was that a firewall connectivity issue with azure from our onpremise servers. But when I tested connection (<storage_account.>blob.core.windows.net:443), it looked fine.

Im not sure if any other ports are required for this to work and the documentation does not mention anything related to the connection requirements. Can some one please help?

Note:

  • I have added azure.client.default.account and azure.client.default.key on all nodes and the way it was added is already verified as working on test VMs outside my firm.
  • I have also restarted the service on all nodes after the above keystore changes

Errors seen in API response and logs

{
  "error": {
    "root_cause": [
      {
        "type": "settings_exception",
        "reason": "Unable to find client with name [default]"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[azure_repo] path  is not accessible on master node",
    "caused_by": {
      "type": "settings_exception",
      "reason": "Unable to find client with name [default]"
    }
  },
  "status": 500
}

[2022-10-17T10:50:12,782][WARN ][r.suppressed             ] [test-cluster-es01] path: /_snapshot/azure_repo, params: {repository=azure_repo}
org.elasticsearch.repositories.RepositoryVerificationException: [azure_repo] path  is not accessible on master node
Caused by: org.elasticsearch.common.settings.SettingsException: Unable to find client with name [default]
        at org.elasticsearch.repositories.azure.AzureStorageService.getClientSettings(AzureStorageService.java:101) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:91) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:624) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:620) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:454) ~[?:?]
        at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:453) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:378) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:103) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:98) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1677) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.0.jar:7.17.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

You should doubleck check if the config is really set in all nodes, your error pretty clear indicates that it is not

[2022-10-17T10:50:12,782][WARN ][r.suppressed ] [test-cluster-es01] path: /_snapshot/azure_repo, params: {repository=azure_repo}
org.elasticsearch.repositories.RepositoryVerificationException: [azure_repo] path is not accessible on master node
Caused by: org.elasticsearch.common.settings.SettingsException: Unable to find client with name [default]

Check the keystore in every node to confirm that the configurations are really there.

Thanks for your reply. However, I have already verified this and double confirmed again.

I ran below steps on all nodes including 'elasticsearch-keystore show' to verify the configuration is set on all nodes.

root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ./elasticsearch-plugin list
ingest-attachment
repository-azure
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# STORAGE_ACCOUNT="elasticbkptest"
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ACCESS_KEY="XXXXXXXXXXXXXXXX/4++ASt15ktPA=="
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin#   
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# echo $STORAGE_ACCOUNT | ./elasticsearch-keystore add -f --stdin azure.client.default.account
SS_KEY | ./elasticsearch-keystore add -f --stdin azure.client.default.key
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# echo $ACCESS_KEY | ./elasticsearch-keystore add -f --stdin azure.client.default.key
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ./elasticsearch-keystore show azure.client.default.account
elasticbkptest
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ./elasticsearch-keystore show azure.client.default.key    
XXXXXXXXXXXXXXXX/4++ASt15ktPA==
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# /etc/init.d/elasticsearch restart
[ ok ] Restarting elasticsearch (via systemctl): elasticsearch.service.
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 

Then ran the repo registration call and still I have this error

root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# curl -k -u elastic -X PUT "https://localhost:9200/_snapshot/azure_backup" -H 'Content-Type: application/json' -d'
> {
>   "type": "azure",
>   "settings": {
>     "container": "testbackup",
>     "chunk_size": "32MB",
>     "compress": true
>   }
> }'
Enter host password for user 'elastic':
{"error":{"root_cause":[{"type":"settings_exception","reason":"Unable to find client with name [default]"}],"type":"repository_verification_exception","reason":"[azure_backup] path  is not accessible on master node","caused_by":{"type":"settings_exception","reason":"Unable to find client with name [default]"}},"status":500}root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# curl -k -u elastic -X POST ""https://localhost:9200/_snapshot/azure_backup/_verify?pretty"
> ^C
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# curl -k -u elastic -X POST "https://localhost:9200/_snapshot/azure_backup/_verify?pretty"
Enter host password for user 'elastic':
{
  "error" : {
    "root_cause" : [
      {
        "type" : "settings_exception",
        "reason" : "Unable to find client with name [default]"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[azure_backup] path  is not accessible on master node",
    "caused_by" : {
      "type" : "settings_exception",
      "reason" : "Unable to find client with name [default]"
    }
  },
  "status" : 500
}

Im afraid if its related to how elasticsearch is installed on the machines inside the firm. We used elasticsearch ansible role for the deployment and the config folder used is non-standard(es_conf_dir as per ansible below). So the keystore file that created for the xpack during deployment is located in a non default path.

Does this affect adding azure repo settings? It doesnt make sense since I am anyway able to list and show the azure repo settings in keystore after adding it though.

Even if we have a custom config location, we should be able to add keystore settings to file in that location - but I can not see such options with the keystore command.

I'm not sure, but I think that the elasticsearch.keystore file needs to be in the ES_PATH_CONF.

Is the elasticsearc.keystore in the same path for ES_PATH_CONF ? The same place where elasticsearch.yml is located.

1 Like

Okay, I fixed this now. I had to export ES_PATH_CONF (since we were using a custom config location) before running keystore commands so that the keystore file matches with what elasticsearch service uses.
Thank you for your time and support :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.