ElasticSearch 7.3 Azure repository snapshot error

[Arun_Yenumula]

1. We are running elasticSearch on 3 nodes (VMs) in Azure inside a docker swarm.
2. Created an Azure storage account for ElasticSearch snapshots.
3. Added Azure storage accountname and key into Elasticsearch-keystore.
4. Running the below curl, receiving a 500 error.

[root@37ff27ca22fb elasticsearch]# bin/elasticsearch-keystore list
keystore.seed

[root@37ff27ca22fb elasticsearch]# AZURE_DEFAULT_ACCOUNT="xxxxxxxxxxx"
[root@37ff27ca22fb elasticsearch]# AZURE_DEFAULT_KEY="yyyyyyyyyy"

[root@37ff27ca22fb elasticsearch]# echo $AZURE_DEFAULT_ACCOUNT | bin/elasticsearch-keystore add --stdin azure.client.default.account
[root@37ff27ca22fb elasticsearch]# echo $AZURE_DEFAULT_KEY | bin/elasticsearch-keystore add --stdin azure.client.default.key

[root@37ff27ca22fb elasticsearch]# bin/elasticsearch-keystore list
azure.client.default.account
azure.client.default.key
keystore.seed

curl -v -u username:password -X PUT 'localhost:9200/_snapshot/my_backup1?pretty' -H 'Content-Type: application/json' -d'{"type":"azure", "settings":{"client":"default"}}'

• About to connect() to localhost port 9200 (#0)
• Trying 127.0.0.1...
• Connected to localhost (127.0.0.1) port 9200 (#0)
• Server auth using Basic with user 'app'
  PUT /_snapshot/my_backup1?pretty HTTP/1.1
  Authorization: Basic YXBwOlV4eTQ0Sk42OTQ5bkF2TFhrM253
  User-Agent: curl/7.29.0
  Host: localhost:9200
  Accept: /

Content-Type: application/json
Content-Length: 49

• upload completely sent off: 49 out of 49 bytes
  < HTTP/1.1 500 Internal Server Error
  < content-type: application/json; charset=UTF-8
  < content-length: 280
  <
  {
  "error" : {
  "root_cause" : [
  {
  "type" : "settings_exception",
  "reason" : "Unable to find client with name [default]"
  }
  ],
  "type" : "settings_exception",
  "reason" : "Unable to find client with name [default]"
  },
  "status" : 500
  }
• Connection #0 to host localhost left intact

Could you provide some direction on where things are wrong?

[Armin_Braun]

@Arun_Yenumula did you restart Elasticsearch after putting the client settings into the key-store? These settings will only be picked-up after a node restart as pointed out by the docs.

[Arun_Yenumula]

@Armin_Braun: Yes, For testing purposes, the client settings have been baked into the elasticsearch-keystore at the time of docker image creation itself. A restart was not required as the container got running with the credentials in place.

1. Are there any other ways to verify if the issue is on Azure side or elasticsearch side for testing?
2. Any sample script to test Azure storage connectivity?

[Armin_Braun]

@Arun_Yenumula

1. Are there any other ways to verify if the issue is on Azure side or elasticsearch side for testing?

I don't think you need to do that. The issue clearly is with your settings not loading properly since the default client is not found, this is an error that's going to be raised before any connection attempt to Azure is made.

Yes, For testing purposes, the client settings have been baked into the elasticsearch-keystore at the time of docker image creation itself

Maybe the issue is somewhere with that. Maybe try the same steps to set up the keystore settings in a shell and then start ES. I just tried your steps locally and it worked just fine, so I'm assuming the issue is with the Docker setup here.