In our search index we want to specify access at very granular level. For eg. Say we have doc1, doc2, doc3 in index and user1 has access to doc1 and doc2, similarly user2 to doc2 and doc3, user3 to doc3 and doc1.
There is no field or pattern that we can use to generalize the query in DLS. The mapping between user and documents are decided at individual user and document level.
The number of users are above 10k and documents in millions. That makes it impractical to store userIds with each document.
One way might be to use a parent-child relationship where the document is the parent and the users allowed to access this document each has a child document. There are drawbacks with parent-child in terms of query performance, query complexity and resource usage, but this might be a case where it is appropriate.
Thanks @Christian_Dahlqvist . Performance is a concern for us. Instead of joins, if we implement terms lookup/level query to lookup another index (which will have the mapping between users - documents) do you think that will have an impact on the performance?
Do you mean running 2 queries - one to get IDs of allowed documents by user and then a query with a potentially huge terms clause limiting access to documents?
Not at application code level. That will probably have an impact because of the huge list going over the network. I was referring to using the below feature in ES - [terms lookup] (Terms query | Elasticsearch Guide [8.4] | Elastic). I haven't tried this yet, not sure if it will help in this scenario.
I have not used that feature, so do not know how performance compares to parent-child. I suspect it comes down to how many documents each user can be associated with (limit seems to be 65536), which you have not provided informtion around.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.