We are trying to use User Assigned Managed Identity (UAMI) for snapshot repository authentication on Elasticsearch v8.17.6 running on Azure Virtual Machines Scale Set (Windows VM).
We followed the official docs:
https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure.html
https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure-client.html
Managed Identity is assigned to the VM and has Storage Blob Data Contributor access to the storage account.
However, we are getting the following error when trying to access the snapshot repository:
"reason": "credential_unavailable_exception: EnvironmentCredential authentication unavailable. Environment variables are not fully configured...
Managed Identity authentication is not available.
...
Azure Powershell authentication failed. Error Details: access denied (\"java.io.FilePermission\" \"<<ALL FILES>>\" \"execute\")"
"POST "https://localhost:9200/_snapshot/elasticsearch-snapshots/_verify?pretty" -k
{
"error" : {
"root_cause" : [
{
"type" : "credential_unavailable_exception",
"reason" : "credential_unavailable_exception: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/environmentcredential/troubleshoot\r\nWorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/java/identity/workloadidentitycredential/troubleshoot\r\nManaged Identity authentication is not available.\r\nSharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.\r\naccess denied (\"java.io.FilePermission\" \"C:\\Windows\\system32\\config\\systemprofile\\AzureToolsForIntelliJ\\AuthMethodDetails.json\" \"read\")\r\naccess denied (\"java.io.FilePermission\" \"<<ALL FILES>>\" \"execute\")\r\nAzure Powershell authentication failed. Error Details: access denied (\"java.io.FilePermission\" \"<<ALL FILES>>\" \"execute\"). To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/powershellcredential/troubleshoot\r\naccess denied (\"java.io.FilePermission\" \"<<ALL FILES>>\" \"execute\")To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azure-identity-java-default-azure-credential-troubleshoot",
"suppressed" : [
{
"type" : "exception",
"reason" : "exception: #block terminated with an error"
}
]
}
],
"type" : "repository_verification_exception",
"reason" : "[elasticsearch-snapshots] path [elasticsearch-snapshots] is not accessible on master node",
"caused_by" : {
"type" : "credential_unavailable_exception",
"reason" : "credential_unavailable_exception: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/environmentcredential/troubleshoot\r\nWorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/java/identity/workloadidentitycredential/troubleshoot\r\nManaged Identity authentication is not available.\r\nSharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.\r\naccess denied (\"java.io.FilePermission\" \"C:\\Windows\\system32\\config\\systemprofile\\AzureToolsForIntelliJ\\AuthMethodDetails.json\" \"read\")\r\naccess denied (\"java.io.FilePermission\" \"<<ALL FILES>>\" \"execute\")\r\nAzure Powershell authentication failed. Error Details: access denied (\"java.io.FilePermission\" \"<<ALL FILES>>\" \"execute\"). To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/powershellcredential/troubleshoot\r\naccess denied (\"java.io.FilePermission\" \"<<ALL FILES>>\" \"execute\")To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azure-identity-java-default-azure-credential-troubleshoot",
"suppressed" : [
{
"type" : "exception",
"reason" : "exception: #block terminated with an error"
}
]
}
},
"status" : 500
}
"
Things we tried:
-
We can successfully fetch the token manually via:
curl -H "Metadata: true" "http://169.254.169.254/metadata/identity/oauth2/token?resource=https://storage.azure.com&api-version=2018-02-01" -
Verified network access to storage container.
-
Added the correct
elasticsearch.ymlconfig:azure.client.default.account: <storage-account-name> azure.client.default.endpoint_suffix: core.windows.net -
Tried with and without
azure.client.default.managed_identity = true(it’s not documented). -
Plugin is bundled, no extra install.
Questions:
- Is there an example of working Managed Identity (MSI) config on Windows VMs for snapshot repo?
- Is there a known issue with the DefaultAzureCredential chain inside Elasticsearch on Windows?
- Should we instead pass the token manually using
elasticsearch-keystore add-file azure.client.default.token_file?
Any guidance would be appreciated!