How to use Azure Storage plugin in ECK for Snapshot

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi All,

I have to configure snapshot in ECK Deployment. Elasticsearch documentation has given example for Google cloud. Please can anyone list the steps for that.

I tried this -
Created a file azure.client.default.credentials_file

{
"azure.client.default.account": "ddddadsdasascs",
"azure.client.default.key": "csvtbdv"
}

Create secret -

kubectl create secret generic azure-snapshot-credentials --from-file=azure.client.default.credentials_file -n dev

and updated the file with this -

spec:
  secureSettings:
  - secretName: "azure-snapshot-credentials"

But I see the pod started crashing with following error -

java.lang.IllegalArgumentException: unknown secure setting [azure.client.default.credentials_file] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

I am pretty sure, i am doing something wrong. Please can someone help me on this.
Thanks

Regards
AK

2

You probably want to install the azure repository plugin.
Please look at ECK plugins documentation.

3

Hi Sebastien @sebgl, I am using custom build image with plugin. Please let me know anything else need to be done.

FROM ${repository_url}/elasticsearch:7.7.0
RUN bin/elasticsearch-plugin install --batch https://<repository_url>/repository-azure-7.7.0.zip

Thanks.

4

Hi Sebgl,

I am able to proceed further via correcting some information. I created a secret with following yaml file.

apiVersion: v1
kind: Secret
metadata:
  name: azure-snapshot-scret
type: Opaque
data:
  azure.client.default.account: <base-64 account name>
  azure.client.default.key: <base-64 key>

Now when i am running repository test following error is coming


{"type": "server", "timestamp": "2020-06-22T15:32:19,515Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "emptydir-els", "node.name": "emptydir-els-es-master-0", "message": "path: /_snapshot/test/_verify, params: {repository=test}", "cluster.uuid": "hZne8eJ5SgOeGeslyLvJkw", "node.id": "0c1AOl99Ta-w-6fE7ivrzA" ,  
"stacktrace": ["org.elasticsearch.common.settings.SettingsException: Invalid azure client settings with name [default]", 
"at org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:112) ~[?:?]", 
"at org.elasticsearch.repositories.azure.AzureStorageService.writeBlob(AzureStorageService.java:327) ~[?:?]", 
"at org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:119) ~[?:?]", 
"at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:101) ~[?:?]", 
"at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:109) ~[?:?]", 
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1065) ~[elasticsearch-7.7.0.jar:7.7.0]", 
"at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:246) ~[elasticsearch-7.7.0.jar:7.7.0]", 
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.7.0.jar:7.7.0]", 
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]", 
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 
"at java.lang.Thread.run(Thread.java:832) [?:?]", 
"Caused by: java.security.InvalidKeyException: Storage Key is not a valid base64 encoded string.", 
"at com.microsoft.azure.storage.StorageCredentials.tryParseCredentials(StorageCredentials.java:68) ~[?:?]", 
"at com.microsoft.azure.storage.CloudStorageAccount.tryConfigureServiceAccount(CloudStorageAccount.java:664) ~[?:?]", 
"at com.microsoft.azure.storage.CloudStorageAccount.parse(CloudStorageAccount.java:285) ~[?:?]", 
"at org.elasticsearch.repositories.azure.AzureStorageService.createClient(AzureStorageService.java:140) ~[?:?]", 
"at org.elasticsearch.repositories.azure.AzureStorageService.buildClient(AzureStorageService.java:117) ~[?:?]", 
"at org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:110) ~[?:?]", 
"... 11 more"] }
5

I am able to fix it now. Values for Account name and Key details after encryption were not proper. Thanks

6

İ did everything accordingly but still failing. I also read a Medium post about this still no result.

7

Probably in some cases I noticed the base64 of key was having a "/n". Make sure proper base64 is being generated.

8

Hi @robinhood_lko,
Can you provide the complete infos to configure the Azure storage? I'm deploying elasticsearch using Helm

So far this is what you have done.
1 - Create a file containing account + key and secret
{
"azure.client.default.account": "ddddadsdasascs",
"azure.client.default.key": "csvtbdv"
}

kubectl create secret generic azure-snapshot-credentials --from-file=azure.client.default.credentials_file -n dev

What was the fix from the exception?

java.lang.IllegalArgumentException: unknown secure setting [azure.client.default.credentials_file] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

2 - What was the fix for the base-64 account & key
3 - Did you set any key in elasticsearch.yml and keystore?
4 - I have installed the repository-azure plugin
5 - Please include any infos that I did not covered above

Regards,
Dis