Im trying to register an azure blob as snapshot repository. I have already tested this with elasticsearch 7.17 on test VMs outside my firm but it is failing when Im trying the same steps inside the firm.
My first impression was that a firewall connectivity issue with azure from our onpremise servers. But when I tested connection (<storage_account.>blob.core.windows.net:443), it looked fine.
Im not sure if any other ports are required for this to work and the documentation does not mention anything related to the connection requirements. Can some one please help?
Note:
I have added azure.client.default.account and azure.client.default.key on all nodes and the way it was added is already verified as working on test VMs outside my firm.
I have also restarted the service on all nodes after the above keystore changes
Errors seen in API response and logs
{
"error": {
"root_cause": [
{
"type": "settings_exception",
"reason": "Unable to find client with name [default]"
}
],
"type": "repository_verification_exception",
"reason": "[azure_repo] path is not accessible on master node",
"caused_by": {
"type": "settings_exception",
"reason": "Unable to find client with name [default]"
}
},
"status": 500
}
[2022-10-17T10:50:12,782][WARN ][r.suppressed ] [test-cluster-es01] path: /_snapshot/azure_repo, params: {repository=azure_repo}
org.elasticsearch.repositories.RepositoryVerificationException: [azure_repo] path is not accessible on master node
Caused by: org.elasticsearch.common.settings.SettingsException: Unable to find client with name [default]
at org.elasticsearch.repositories.azure.AzureStorageService.getClientSettings(AzureStorageService.java:101) ~[?:?]
at org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:91) ~[?:?]
at org.elasticsearch.repositories.azure.AzureBlobStore.getAzureBlobServiceClientClient(AzureBlobStore.java:624) ~[?:?]
at org.elasticsearch.repositories.azure.AzureBlobStore.asyncClient(AzureBlobStore.java:620) ~[?:?]
at org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:454) ~[?:?]
at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45) ~[?:?]
at org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:453) ~[?:?]
at org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:378) ~[?:?]
at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:103) ~[?:?]
at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:98) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1677) ~[elasticsearch-7.17.0.jar:7.17.0]
at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392) ~[elasticsearch-7.17.0.jar:7.17.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.0.jar:7.17.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.0.jar:7.17.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
You should doubleck check if the config is really set in all nodes, your error pretty clear indicates that it is not
[2022-10-17T10:50:12,782][WARN ][r.suppressed ] [test-cluster-es01] path: /_snapshot/azure_repo, params: {repository=azure_repo}
org.elasticsearch.repositories.RepositoryVerificationException: [azure_repo] path is not accessible on master node
Caused by: org.elasticsearch.common.settings.SettingsException: Unable to find client with name [default]
Check the keystore in every node to confirm that the configurations are really there.
Im afraid if its related to how elasticsearch is installed on the machines inside the firm. We used elasticsearch ansible role for the deployment and the config folder used is non-standard(es_conf_dir as per ansible below). So the keystore file that created for the xpack during deployment is located in a non default path.
Does this affect adding azure repo settings? It doesnt make sense since I am anyway able to list and show the azure repo settings in keystore after adding it though.
Even if we have a custom config location, we should be able to add keystore settings to file in that location - but I can not see such options with the keystore command.
Okay, I fixed this now. I had to export ES_PATH_CONF (since we were using a custom config location) before running keystore commands so that the keystore file matches with what elasticsearch service uses.
Thank you for your time and support
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.