Getting maxClauseCount error when using document-level security

Elastic Stack Elasticsearch
1

We tag our documents with security ids and wish to filter the documents at query time with a list of security ids associated with the user.

We added a field to user.metadata called 'sids' which is an array of integers. Our document has a field called 'securityIds' that is an array of integers. We set up the role's query as follows:

   "query": {
      "template":{
           "inline": "{\"terms\" : { \"securityIds\" : {{#toJson}}_user.metadata.sids{{/toJson}} } }"
      }
    }

This works perfectly until we add more than 1024 sids to the user. At that point we get the following error:

caused by: too_many_clauses: maxClauseCount is set to 1024
org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:136)
org.elasticsearch.index.query.TermsQueryBuilder.handleTermsQuery(TermsQueryBuilder.java:450)
org.elasticsearch.index.query.TermsQueryBuilder.doToQuery(TermsQueryBuilder.java:411)
org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97)
org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:306)
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:323)
org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:305)
org.elasticsearch.xpack.security.authz.accesscontrol.SecurityIndexSearcherWrapper.wrap(SecurityIndexSearcherWrapper.java:176)
org.elasticsearch.index.shard.IndexSearcherWrapper.wrap(IndexSearcherWrapper.java:75)

Note that if we implement this w/o xpack by passing in a filter list into the query containing the sid list, the queries work fine. But we were hoping that xpack would give us better performance than passing large fiters into every query.

Is there any way to accomplish this with xpack?

Also any thoughts you may have on the performance of xpack for this kind of usage would be greatly appreciated.

Thanks!

2

What is the relationship between the number of sids and documents?

Note that if we implement this w/o xpack by passing in a filter list into the query containing the sid list, the queries work fine

To be clear, are you talking about something like:

{
  "query": { 
    "bool": { 
      "filter": [ 
        { "terms":  { "securityIds": "..." }}
      ]
    }
  }
}
3

Yes, exactly.

4

Thanks for bringing this up; we're planning to fix this in our next minor release.

Regarding performance, I think it will depend on the relationship between security ids, documents, and users.

5

Great, thanks.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.