Use roles instead of ApiKey for Search Applications

[Artem_Ruzak]

Hello Elastic community,

I'm looking to implement a scenario where a client (the user's browser) makes API requests directly to the Elasticsearch service, allowing the client to access the Elasticsearch cluster directly.

I came across the Elastic Search Applications documentation, which seems to fit my needs well. My plan is to define a search template where the front end only needs to pass specific search parameters, and Elasticsearch then returns the search results directly to the client.

However, I’m encountering challenges with security configuration. According to the documentation, accessing a search application requires an API key, which would need to be included in requests from the front end, thereby exposing it to users.

We are currently using SSO with OpenID, so all our users are logged in, and I can perform role mapping to assign them specific roles. Ideally, I’d like to configure permissions to allow access to particular search applications based on custom roles, mapping these roles to users upon login.

My question is whether this type of configuration is possible, and if so, how to set it up. Any guidance or resources would be greatly appreciated.

Thank you!

[ashishtiwari1993 (Ashish Tiwari)]

Hello @Artem_Ruzak, Welcome back to the Elastic Community.

Yes, as mentioned in above link, you can set role restrictions, Document level security or field level security. This will allow you to assign granular level access to your users.

[Artem_Ruzak]

Hello @ashishtiwari1993 ,
thank you for replay. I already had a look on the role restrictions, but they seemed to be hard to use in my case - the role restriction are applied to API keys, which means I anyway had to have API key created and it will be visible in frontend communication - and this is not the scenario I am looking for.

I want to have users logged over SSO, use role mapping to assign specific role to the user and I hoped that I can inside the role restrict user to use only search application to search data directly from frontend.

I tried to create role and add role restriction there, but it does not work:

POST /_security/role/ar_appsearch
{
    "cluster": [],
    "indices": [
      {
        "names": [
          "idx-ocation",
          "idx-Location2"
        ],
        "privileges": [
          "read"
        ],
        "allow_restricted_indices": false
      }
    ],
    "applications": [],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    },
    "restriction": {
      "workflows": [
        "search_application_query"
      ]
    }
  }

got an error:

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "failed to parse role [ar_appsearch]. unexpected field [restriction]"
      }
    ],
    "type": "parse_exception",
    "reason": "failed to parse role [ar_appsearch]. unexpected field [restriction]"
  },
  "status": 400
}

[Sean_Story]

Hi @Artem_Ruzak ,

I wrote a blog that kind of gets into this, and provides an example architecture: Adding document level security (DLS) to your internal knowledge search - Search Labs

If you don't want the browser sending requests directly to Elasticsearch (TOTALLY valid), then you need to have your own backend between the browser and Elasticsearch, and you'll need to have your browser excuting search reqeusts instead to your backend, which can then pick the API key/role descriptor that matches your authenticated end user, and execute the request to Elasticsearch with the corresponding API key.

I tried to create role and add role restriction there, but it does not work:

What version of Elasticsearch are you on? The Restriction feature was added in 8.9: Role restriction | Elasticsearch Guide [8.9] | Elastic

[Artem_Ruzak]

@Sean_Story

Adding document level security (DLS) to your internal knowledge search - Search Labs

Than you for response @Sean_Story !
In my use case I want indeed fronend to talk directly with elsticsearch and perform simple searches/aggreagations directly.

I red the blog you mentioned and as I understood from your implemenation, the solution is to create backend service which will return API key with validity of 1 hour, frontend will request this APIKey and use it when calling the elastic for some _search

I think it is working, but cumbersome solution. We need to implement additional service to generate user keys and we need to set some short validity interval for such API Keys. Plus, it means we would also need to implement some ApiKey refresh mechanism and monitor key validity.

In would be much easier for me if can set it in role and user role mapping for assignment of roles to user during authorisation process. It will also work with document-level security.

The only trouble I have right now -- it is not possible to define in role definition configuration like this:

  "restriction": {"workflows": ["search_application_query"]},

Or am I misisng something?

@Sean_Story

What version of Elasticsearch are you on?

We are using elastic cloud, currently on 8.15.3 version

[Sean_Story]

@Artem_Ruzak

the role restriction are applied to API keys, which means I anyway had to have API key created and it will be visible in frontend communication - and this is not the scenario I am looking for.

@Artem_Ruzak

In my use case I want indeed fronend to talk directly with elsticsearch and perform simple searches/aggreagations directly.

Hi @Artem_Ruzak - these two statements seem at odds with one another. I'd assumed from the first that you didn't want the browser talking directly to Elasticsearch because you didn't want to the browser to be directly aware of any Elasticsearch credentials (totally valid). But from the second message it seems that you do actually expect the browser to be able to communicate with Elasticsearch, in which case you will need to have some Elasticsearch credential accessible by your frontend. Am I misunderstanding your requirements?

@Artem_Ruzak

I think it is working, but cumbersome solution. We need to implement additional service to generate user keys and we need to set some short validity interval for such API Keys. Plus, it means we would also need to implement some ApiKey refresh mechanism and monitor key validity.

Indeed. There is a tradeoff here between security and convenience. The shorter the lifetime of the credential, the more secure, but also the more work you need to do to ensure that the credentials keep refreshed.

@Artem_Ruzak

In would be much easier for me if can set it in role and user role mapping for assignment of roles to user during authorisation process. It will also work with document-level security.

The only trouble I have right now -- it is not possible to define in role definition configuration like this:

Ah! Ok, I'd not realized that our restriction feature is currently limited to API keys only, and can't be used on concrete roles. See here:

Note: Currently, the role restriction is only supported for API keys, with limitation that the API key can only have a single role descriptor.

I'll reach out to our team that maintains Search Applications to see if they have any thoughts. But in the mean time, you've got a few options:

• embrace the struggle, and use API keys with the workflow restriction for Search Applications
• don't use Search Applications, and instead hit Elasticsearch index APIs directly (avoids needing the workflow restriction)
• build a more complex backend to sit between your browser and your Elasticsearch, which can use longer-lived API keys for search, and apply end-user DLS filters at query time.

[Artem_Ruzak]

Hello Sean,

Thank you for your feedback. I realize I may not have been entirely clear earlier, so let me summarize our current setup and the challenges we’re addressing:

1. Current Setup:
   We are using ElasticCloud with SSO for user authentication. Once authenticated, we employ role mappings to assign specific roles to users based on the content of their SSO ticket. These roles control access to data by restricting visibility to certain indices and Kibana spaces.

2. Data Retrieval:
   To retrieve data from Elasticsearch, we’ve built a series of backend APIs. Each API follows a similar structure:

• Authenticate the user.
• Perform a search query on Elasticsearch.
• Return the results to the frontend.
  It is classic, but involves creating multiple wrappers around the _search query, which adds complexity and slows down both development and execution.

3. Exploring Search Applications:
   I'm currently exploring the use of search applications as an alternative to our backend APIs. The main advantages we want to achieve:

• Reducing the need for redundant API wrappers.
• Improving query execution speed.
• Improving development/changes speed

From security/load protection point of view, search applications allow us to limit the types of queries users can run, define search parameters, and add parameter validation — all of which help prevent misuse or malicious activity.

Roadblock:
The primary challenge with this approach is access control using ApiKeys. While ApiKeys can work (with a mechanism for frequent key refreshes and short validity periods), the ideal solution would be to enforce these restrictions through roles.

If we could define all necessary permissions within roles and continue leveraging role mapping via SSO, it would eliminate the need to expose ApiKeys. This way, we could:

• Maintain a secure, keyless setup.
• Use SSO tokens and Elastic’s security features to ensure controlled and safe searches defining it all in the Elasticsearch .
• Enable the frontend to interact directly with Elasticsearch without compromising security.

I would be glad to hear if this is possible to add restriction feature to the role definitions, everything else is already in place.

Best regards, Artem

[ioana.tagirta]

It is not currently possible to use restriction with role definitions, but the arguments you are making are compelling.
I cannot make any promises for when restriction will be supported with Elasticsearch roles.
Search applications fit very well with the setup you are describing and I agree that managing API keys has its own challenges.
I don't have a good answer, just a suggestion that you might want to consider having for now a backend service that manages API keys, that clients can call to retrieve an API key. This way you would still remove the backend that wraps _search calls.

[Itamar_Syn-Hershko]

For plenty of reasons (security by obscurity, letting your FE do Elasticsearch DSL calls and be aware of DB models, and more) - Don't do it.

[Artem_Ruzak]

Hi,
Have a closer look at Elastic Search Applications documentation - it is exactly aimed to hide complexity of DSL , indexes and other stuff from the front end user, provide parameter validation and ways to perform safe search.
I am curious to discuss specific reasons you might have.

Best regards, Artem

[Artem_Ruzak]

Hello @Sean_Story, @ioana.tagirta ,

I happy you found my request reasonable -- in order to get this thing moving -- shall I create feqture request or anything elase on Elastic?

Best regards, Artem

[Sean_Story]

Hi @Itamar_Syn-Hershko,

While you're right to some extent in the general case, as Artem points out, Search Applications are designed specifically for this use case.

[Sean_Story]

@Artem_Ruzak

in order to get this thing moving -- shall I create feqture request or anything elase on Elastic?

Please do! If you have a support relationship with Elastic already, your support/sales contact should be able to help you file an Enhancement Request. Otherwise, I'd encourage you to just file a github issue at GitHub - elastic/elasticsearch: Free and Open Source, Distributed, RESTful Search Engine and link this discuss forum thread.

[Artem_Ruzak]

Hi,
Thank you for your help!
We are paying elastic cloud clients. I will reach thru support and make sure we have registered feature request there.

Best regards, Artem