Unable to create snapshot repo in azure

jaykb77 · October 17, 2022, 2:49pm

Im trying to register an azure blob as snapshot repository. I have already tested this with elasticsearch 7.17 on test VMs outside my firm but it is failing when Im trying the same steps inside the firm.

My first impression was that a firewall connectivity issue with azure from our onpremise servers. But when I tested connection (<storage_account.>blob.core.windows.net:443), it looked fine.

Im not sure if any other ports are required for this to work and the documentation does not mention anything related to the connection requirements. Can some one please help?

Note:

I have added azure.client.default.account and azure.client.default.key on all nodes and the way it was added is already verified as working on test VMs outside my firm.
I have also restarted the service on all nodes after the above keystore changes

Errors seen in API response and logs

{
  "error": {
    "root_cause": [
      {
        "type": "settings_exception",
        "reason": "Unable to find client with name [default]"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[azure_repo] path  is not accessible on master node",
    "caused_by": {
      "type": "settings_exception",
      "reason": "Unable to find client with name [default]"
    }
  },
  "status": 500
}

[2022-10-17T10:50:12,782][WARN ][r.suppressed             ] [test-cluster-es01] path: /_snapshot/azure_repo, params: {repository=azure_repo}
org.elasticsearch.repositories.RepositoryVerificationException: [azure_repo] path  is not accessible on master node
Caused by: org.elasticsearch.common.settings.SettingsException: Unable to find client with name [default]
        at org.elasticsearch.repositories.azure.AzureStorageService.getClientSettings(AzureStorageService.java:101) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:91) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:624) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:620) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:454) ~[?:?]
        at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:453) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:378) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:103) ~[?:?]
        at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:98) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1677) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.0.jar:7.17.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

leandrojmp · October 17, 2022, 6:38pm

You should doubleck check if the config is really set in all nodes, your error pretty clear indicates that it is not

[2022-10-17T10:50:12,782][WARN ][r.suppressed ] [test-cluster-es01] path: /_snapshot/azure_repo, params: {repository=azure_repo}
org.elasticsearch.repositories.RepositoryVerificationException: [azure_repo] path is not accessible on master node
Caused by: org.elasticsearch.common.settings.SettingsException: Unable to find client with name [default]

Check the keystore in every node to confirm that the configurations are really there.

jaykb77 · October 18, 2022, 6:44am

Thanks for your reply. However, I have already verified this and double confirmed again.

I ran below steps on all nodes including 'elasticsearch-keystore show' to verify the configuration is set on all nodes.

root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ./elasticsearch-plugin list
ingest-attachment
repository-azure
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# STORAGE_ACCOUNT="elasticbkptest"
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ACCESS_KEY="XXXXXXXXXXXXXXXX/4++ASt15ktPA=="
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin#   
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# echo $STORAGE_ACCOUNT | ./elasticsearch-keystore add -f --stdin azure.client.default.account
SS_KEY | ./elasticsearch-keystore add -f --stdin azure.client.default.key
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# echo $ACCESS_KEY | ./elasticsearch-keystore add -f --stdin azure.client.default.key
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ./elasticsearch-keystore show azure.client.default.account
elasticbkptest
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# ./elasticsearch-keystore show azure.client.default.key    
XXXXXXXXXXXXXXXX/4++ASt15ktPA==
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# /etc/init.d/elasticsearch restart
[ ok ] Restarting elasticsearch (via systemctl): elasticsearch.service.
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin#

Then ran the repo registration call and still I have this error

root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# curl -k -u elastic -X PUT "https://localhost:9200/_snapshot/azure_backup" -H 'Content-Type: application/json' -d'
> {
>   "type": "azure",
>   "settings": {
>     "container": "testbackup",
>     "chunk_size": "32MB",
>     "compress": true
>   }
> }'
Enter host password for user 'elastic':
{"error":{"root_cause":[{"type":"settings_exception","reason":"Unable to find client with name [default]"}],"type":"repository_verification_exception","reason":"[azure_backup] path  is not accessible on master node","caused_by":{"type":"settings_exception","reason":"Unable to find client with name [default]"}},"status":500}root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# 
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# curl -k -u elastic -X POST ""https://localhost:9200/_snapshot/azure_backup/_verify?pretty"
> ^C
root@test-search-es01.qaextranet:/a/elasticsearch/elasticsearch-current/bin# curl -k -u elastic -X POST "https://localhost:9200/_snapshot/azure_backup/_verify?pretty"
Enter host password for user 'elastic':
{
  "error" : {
    "root_cause" : [
      {
        "type" : "settings_exception",
        "reason" : "Unable to find client with name [default]"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[azure_backup] path  is not accessible on master node",
    "caused_by" : {
      "type" : "settings_exception",
      "reason" : "Unable to find client with name [default]"
    }
  },
  "status" : 500
}

jaykb77 · October 18, 2022, 1:14pm

Im afraid if its related to how elasticsearch is installed on the machines inside the firm. We used elasticsearch ansible role for the deployment and the config folder used is non-standard(es_conf_dir as per ansible below). So the keystore file that created for the xpack during deployment is located in a non default path.

Does this affect adding azure repo settings? It doesnt make sense since I am anyway able to list and show the azure repo settings in keystore after adding it though.

Even if we have a custom config location, we should be able to add keystore settings to file in that location - but I can not see such options with the keystore command.

github.com

elastic/ansible-elasticsearch/blob/main/tasks/xpack/security/elasticsearch-security.yml

---
#Security specific configuration done here

#TODO: 1. Skip users with no password defined or error 2. Passwords | length > 6

#-----------------------------Create Bootstrap User-----------------------------------
### START BLOCK elasticsearch keystore ###
- name: create the elasticsearch keystore
  block:
  - name: create the keystore if it doesn't exist yet
    become: yes
    command: >
     {{es_home}}/bin/elasticsearch-keystore create
    args:
      creates: "{{ es_conf_dir }}/elasticsearch.keystore"
    environment:
      ES_PATH_CONF: "{{ es_conf_dir }}"

  - name: Check if bootstrap password is set
    become: yes

This file has been truncated. show original

leandrojmp · October 18, 2022, 1:26pm

I'm not sure, but I think that the elasticsearch.keystore file needs to be in the ES_PATH_CONF.

Is the elasticsearc.keystore in the same path for ES_PATH_CONF ? The same place where elasticsearch.yml is located.

jaykb77 · October 18, 2022, 1:33pm

Okay, I fixed this now. I had to export ES_PATH_CONF (since we were using a custom config location) before running keystore commands so that the keystore file matches with what elasticsearch service uses.
Thank you for your time and support

system · November 15, 2022, 1:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.