We are using ElasticSearch 6.5.4 and recently implemented Document Level security on all indexes (prior to this we were using our own internal custom developed solution). Since going live we see increased utilization of "Fixed Bitsets" on all indexes, to the point of heap exhaustion if we do not periodically (like twice a day) clear the request cache. This issue did not occur prior to implementing document level filtering. Any thoughts?
How many nodes & JVM heap size? How many documents do you have in these indices? How many unique roles are using different document level security queries?
The fixed bitsets are used to keep a cache of authorized documents for a given role (you can have different roles, but if they map to the same underlying query for the document level security - this would only be one).
We have a couple dozen nodes, most with 32G heap. The indices often reach single-digit billions of docs. There's only one role using document-level security, which reads from the user's metadata to enforce access.
Though there's only one role, I wonder whether the user's metadata gets included as part of the key when caching FixedBitSets – i.e. does each user with differing metadata (and thus differing rendered document-level security queries) get its own distinct FixedBitSet?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.