Use azure active directory with NEST/Elasticsearch.net

Elastic Stack Elasticsearch
1

We are working on a project where we are authenticating users with Azure Active Directory. Upon the successful authentication, the user's browsers receive an Id and Access token, and then we use the same access token to query other Microsoft products (Sharepoint, OneDrive, etc).

We are planning to use Elastic Search for our search need. We have already set up SAML/ OpenId realms on our ECE Deployment Portal and Cluster. So if any users try to access ECE deployment portal/ Kibana, they will be prompted to authenticate against Microsoft Azure AD, and upon successful authentication, they get redirected to ECE or Kibana.

We are using C# and NEST dll (ElasticSearch.Net) to create queries and search the elastic search endpoint. We are not sure how exactly should we use the access token received on the UI side with Elastic Search to query out indices. We know, we can use native user credentials or API keys to access the elastic search but we want to use the same azure ad authentication flow(SAML/OpenID) to access Elastic Search.

Is it possible to use the Azure AD access token received on the UI side to access & query Elastic Search Clusters or is there any other way to re-authenticate users while they try to access Elastic Search Cluster?
Is there a way to authenticate the users with the elastic search endpoint and generate an access token that can be used to query elastic search further?

In short, we want to re-authenticate users with Elastic Search while querying the data?

var settings = new ConnectionSettings(new Uri(mEsQuerySource.Url));
settings.BasicAuthentication("user", "plain text password");
mClient = new ElasticClient(settings);
2

In current versions of Elasticsearch (as I write this, 7.14 is the latest version) there is no way to use an Azure AD access token to directly access Elasticsearch.
That is, you cannot have your application authenticate directly to AAD and then use the tokens you receive from AAD as a credential to authenticate to Elasticsearch.
There is no authentication provider in Elasticsearch that works with arbitrary tokens from an external issuer.

You can however do the same thing that ECE and Kibana do and perform SAML or OpenID Connect authentication via Elasticsearch, in order to generate Elasticsearch access & refresh tokens (which are separate from the Azure AD tokens).

There is documentation on how to perform SAML or OIDC authentication to Elasticsearch via a custom application.

The high level overview would be (I assume SAML here, but OIDC would be similar):

  • When a user accesses your application they would authenticate against AzureAD as normal
  • Then, you would use the Elasticsearch APIs to perform an additional authentication against an Elasticsearch SAML realm with Elasticsearch as the service provider and AzureAD as the Identity Provider.
  • Since the user is already authenticated within Azure AD, that second authentication process should be transparent to the user - AAD will simply issue a new SAML assertion with Elasticsearch as the recipient.
  • Those Elasticsearch APIs will accept the SAML assertion, and return a pair of tokens (access + refresh) that can be used to authenticate to Elasticsearch
  • Your application will retain the access + refresh tokens for the user's session
  • The access token will be used to authenticate when accessing Elasticsearch APIs
  • The refresh token will be used to generate a new access token when the old one expires (or is about to expire).

If your users are in an identity store that Elasticsearch can query (e.g. something that supports LDAP search), then another option is to use the Elasticsearch run-as capability.
In this case your application would authenticate to Elasticsearch using a single system credential (probably a user in the native realm). That user would have permission to run-as all other users and this can be used to perform searches on behalf of your end users without needing them to authenticate directly to Elasticsearch.

The final option would be to implement a custom realm, if you have engineers who are comfortable writing Java.

3

Hi Tim,

Thanks for responding!

I do have a follow-up question on point 2 but first, let me explain how point 1 works in our case -

  • When a user accesses your application they would authenticate against AzureAD as normal
    We are using C# and Angular on the front end and we would be using MSAL libraries to authenticate our users. For authentication, we are opening the Microsoft Authentication page for users so that they can use their Azure AD credentials to authenticate themselves.

  • Then, you would use the Elasticsearch APIs to perform an additional authentication against an Elasticsearch SAML realm with Elasticsearch as the service provider and AzureAD as the Identity Provider.
    When you say we would need to use the Elasticsearch APIs to perform an additional authentication, do you mean we will have to redirect the user to the Microsoft Authentication page again?

When I use a service account to prepare my realm for user authentication and execute the below request, It returns a redirect URL and the documentation says we need to redirect our application to this URL for authentication to happen.

POST /_security/oidc/prepare
{
  "realm" : "oidc1"
}

{
  "redirect" : "https://login.microsoftonline.com/{tenant-id}/oauth2/v2.0/authorize?scope=openid+email&response_type=code&redirect_uri=https://xyz.azure.elastic.bain.com%3A9243%2Fapi%2Fsecurity%2Foidc%2Fcallback&state=zoTG-Iaj20hS9ag3TbRlae6woNtPv5mE1Elku6q5UVg&nonce=Y-BkrE2IE2LZ5MgKzKJZIZENrWdDAMty1DlHPk7bV5w&client_id={client_id}",
  "state" : "zoTG-Iaj20hS9ag3TbRlae6woNtPv5mE1Elku6q5UVg",
  "nonce" : "Y-BkrE2IE2LZ5MgKzKJZIZENrWdDAMty1DlHPk7bV5w",
  "realm" : "oidc1"
}

You see, we have already redirected our users (on point 1) to authenticate themselves and if we follow this documentation we would have to redirect the users again. It would not be a very good user experience if we ask them to authenticate themselves twice for the same application.

Is there a way to authenticate users against the elastic realm without redirecting them again to the Microsoft authentication page as we have already authenticated the user (in point 1)?

Thanks in advance!

4

Yes, that's exactly what I mean.
However, per my earlier comment:

Since the user is already authenticated within Azure AD, that second authentication process should be transparent to the user - AAD will simply issue a new SAML assertion with Elasticsearch as the recipient.

Depending on exactly how AAD is configured (sorry, I'm not an expert on the specific of AAD configuration, so I can't be more precise than that), the user is unlikely to even see the a login page. Because they have a valid AAD session, they don't need to enter their credentials again, we simply need to trigger a SAML assertion with the necessary details to be sent to Elasticsearch.

Yes, that is exactly what you need to do. Elasticsearch cannot use your existing tokens because they have nothing to do with Elasticsearch. You need to generate new ones.

You have, but Elasticsearch hasn't.

5

Hi Tim,

Following your suggestions and elastic documentation from OIDC, I registered two azure ad applications (1 for custom web app and 1 for elastic search realm setup) on my tenant.

Here are my realm configurations -

xpack:
  security:
    authc:
      realms:
        oidc:
          oidc1:
            order: 2
            rp.client_id: "{client-id}"
            rp.response_type: "code"
            rp.requested_scopes: ["openid", "email"]
            rp.redirect_uri: "https://customwebapp.azurewebsites.net"
            op.issuer: "https://login.microsoftonline.com/{tenanet-id}/v2.0"
            op.authorization_endpoint: "https://login.microsoftonline.com/{tenanet-id}/oauth2/v2.0/authorize"
            op.token_endpoint: "https://login.microsoftonline.com/{tenanet-id}/oauth2/v2.0/token"
            op.userinfo_endpoint: "https://graph.microsoft.com/oidc/userinfo"
            op.endsession_endpoint: "https://login.microsoftonline.com/{tenanet-id}/oauth2/v2.0/logout"
            rp.post_logout_redirect_uri: "https://customwebapp.azurewebsites.net"
            op.jwkset_path: "https://login.microsoftonline.com/{tenanet-id}/discovery/v2.0/keys"
            claims.principal: email
            claims.groups : groups

I am able to prepare the OIDC realm and getting the redirect, state, and nonce values as a response from elastic. When I redirect the browser to the redirect URL, it asks me to authenticate with Microsoft azure ad. It does ask me to re-authenticate but doesn't ask for a password as I am already logged in to the browser session. Upon successful authentication, it returns the browser window to my web app.

Here is how I am doing the prepare api call from my angular app -

PrepElastic() {
    var url = "https://{appid}.eastus2.azure.elastic.xyz.com:9243/_security/oidc/prepare"
    let authorizationData = 'Basic ' + btoa('facilitator' + ':' + 'somePasswordHere');

    const headers = { 'Authorization': authorizationData, 'Content-Type': 'application/json' };
    const body = { "realm": "oidc1" };
    return this.http.post<any>(url, body, { headers });
  }

I store the state and nonce returned from the prepare call and use the same to authenticate with authenticate API but it gives me an error in the response.

AuthenticateElastic() {
    var url = "https://{appid}.eastus2.azure.elastic.xyz.com:9243/_security/oidc/authenticate"
    let authorizationData = 'Basic ' + btoa('facilitator' + ':' + 'somePasswordHere');

    const headers = { 'Authorization': authorizationData, 'Content-Type': 'application/json' };
    const body = {
      "redirect_uri": this.dataShareService.getAuthCodeUrl(),
      "state": this.dataShareService.getstate(),
      "nonce": this.dataShareService.getnonce(),
      "realm": "oidc1"
    };
    return this.http.post<any>(url, body, { headers });
  }

Here is the error message I am getting on the browser side in response -

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [<OIDC Token>] for action [cluster:admin/xpack/security/oidc/authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [<OIDC Token>] for action [cluster:admin/xpack/security/oidc/authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

I turned on tracing on elastic search realm in order to find out the actual error -

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.security.authc.oidc": "trace"
  }
}

and I see below error message logged-

[elasticsearch.server][WARN] Authentication to realm oidc1 failed - Failed to authenticate user with OpenID Connect (Caused by ElasticsearchSecurityException[Failed to validate the response, the response did not contain a state parameter])

Reading the above message seems that we are not passing state but I can assure that we are passing all the correct values with the authenticate request.

image
image1604×179 8.61 KB

Could you please help here?

Thanks,
Abhishek

6

So the problem was with the Auth-Code redirect URI returned from prepare call after the authentication. It has session_id appended to it at the end. Once I remove it, it worked. Thanks Tim for the support.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.