# Match array of terms in filtered query **URL:** \ **Category:** Elasticsearch\ **Created:** [August 24, 2010, 5:43pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250 "2010-08-24T17:43:39Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Richard\_Livsey](https://avatars.discourse-cdn.com/v4/letter/r/dc4da7/32.png) [@Richard\_Livsey](https://discuss.elastic.co/u/Richard_Livsey)\ **Post date:** [August 24, 2010, 5:43pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/1 "2010-08-24T17:43:39Z")
Each user in our app can see a subset of the documents based on a number of conditions (permissions etc...) and I'm wondering what the best approach to building a restricted query would be. Andrei asked a similar question about restricting the query with boolean vs filters in which filters came up as the most efficient method: [http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865](http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865) However in our data model we don't have a single field which we can filter by, it's more a case of getting a list of all the IDs of documents the user can access and then searching them. Eg, modelled as a boolean query, which works but isn't very elegant and I can see it having issues with performance as the query grows: ``` { :query => { :bool => { :must => [ { :query_string => { :query => params[:q] } }, { :query_string => { :default_field => "id", :query => accessible_ids.join(" OR ") } } ] } } } ``` Is there a way of performing a term query against an array of terms so that I could convert this to a filter? Or is there a better way of doing this that someone knows of? Thanks in advance. -- Richard Livsey Minutebase - Online Meeting Minutes [http://minutebase.com](http://minutebase.com) [http://livsey.org](http://livsey.org) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 24, 2010, 5:53pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/2 "2010-08-24T17:53:12Z")
There is the terms filter which should fit better in this case. Can you explain more why you get all the ids and then do a search? How do you get all the ids? Can't that be "logical" groups that are added to each doc, and filter by that? It will be much more efficient. -shay.banon On Tue, Aug 24, 2010 at 8:43 PM, Richard Livsey [livsey@gmail.com](mailto:livsey@gmail.com) wrote: > Each user in our app can see a subset of the documents based on a > number of conditions (permissions etc...) and I'm wondering what the > best approach to building a restricted query would be. > > Andrei asked a similar question about restricting the query with > boolean vs filters in which filters came up as the most efficient > method: > > [http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865](http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865) > > However in our data model we don't have a single field which we can > filter by, it's more a case of getting a list of all the IDs of > documents the user can access and then searching them. > > Eg, modelled as a boolean query, which works but isn't very elegant > and I can see it having issues with performance as the query grows: > > ``` > { > :query => { > :bool => { > :must => [ > { > :query_string => { > :query => params[:q] > } > }, > { > :query_string => { > :default_field => "id", > :query => accessible_ids.join(" OR ") > } > } > ] > } > } > } > > ``` > > Is there a way of performing a term query against an array of terms so > that I could convert this to a filter? > Or is there a better way of doing this that someone knows of? > > Thanks in advance. > > -- > Richard Livsey > Minutebase - Online Meeting Minutes > [http://minutebase.com](http://minutebase.com) > [http://livsey.org](http://livsey.org) ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [August 24, 2010, 5:56pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/3 "2010-08-24T17:56:42Z")
Hi Richard, I have a similar situation where for any user we construct a query string that enforces what they are entitled to. This query string can be arbitrarily complex. Just taking that query string and wrapping it in a filter seems to work very well. So, my final query typically ends up looking like: ``` 'filtered' : { 'query' : { 'query_string' : {'query':mainquery} }, 'constant_score' : { 'filter' : { 'query' : { 'query_string' : {'query':userentitlements} } } } } ``` So you can keep doing similar to what you are doing, but this query structure yields good caching on the user entitlements part. Not sure if this is optimal, but is working quite well in my performance testing thus far. Regards, Paul On Aug 24, 11:43 am, Richard Livsey [liv...@gmail.com](mailto:liv...@gmail.com) wrote: > Each user in our app can see a subset of the documents based on a > number of conditions (permissions etc...) and I'm wondering what the > best approach to building a restricted query would be. > > Andrei asked a similar question about restricting the query with > boolean vs filters in which filters came up as the most efficient > method:[http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filt](http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filt)... > > However in our data model we don't have a single field which we can > filter by, it's more a case of getting a list of all the IDs of > documents the user can access and then searching them. > > Eg, modelled as a boolean query, which works but isn't very elegant > and I can see it having issues with performance as the query grows: > > ``` > { > :query => { > :bool => { > :must => [ > { > :query_string => { > :query => params[:q] > } > }, > { > :query_string => { > :default_field => "id", > :query => accessible_ids.join(" OR ") > } > } > ] > } > } > } > > ``` > > Is there a way of performing a term query against an array of terms so > that I could convert this to a filter? > Or is there a better way of doing this that someone knows of? > > Thanks in advance. > > -- > Richard Livsey > Minutebase - Online Meeting Minuteshttp://minutebase.comhttp://livsey.org ---
**Author:** ![Richard\_Livsey\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richard_livsey_2/32/3241_2.png) [@Richard\_Livsey\_2](https://discuss.elastic.co/u/Richard_Livsey_2)\ **Post date:** [August 24, 2010, 5:59pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/4 "2010-08-24T17:59:48Z")
I wish it was simpler, but there's no real grouping of documents on a per-user basis. A user can see a document if any of the following are true (and a few other conditions): - they are an admin - they created the document - they are a member of the project the document belongs to - the document is "shared" with them - the document is public - etc... The query to generate this list of documents can be fairly hairy, so we cache a list of document IDs on a per-user basis. So when doing a full-text search I have the IDs to restrict against, but unfortunately it could number in the hundreds. Thanks. On Tue, Aug 24, 2010 at 6:53 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > There is the terms filter which should fit better in this case. Can you > explain more why you get all the ids and then do a search? How do you get > all the ids? Can't that be "logical" groups that are added to each doc, and > filter by that? It will be much more efficient. > -shay.banon > > On Tue, Aug 24, 2010 at 8:43 PM, Richard Livsey [livsey@gmail.com](mailto:livsey@gmail.com) wrote: > > > Each user in our app can see a subset of the documents based on a > > number of conditions (permissions etc...) and I'm wondering what the > > best approach to building a restricted query would be. > > > > Andrei asked a similar question about restricting the query with > > boolean vs filters in which filters came up as the most efficient > > method: > > > > [http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865](http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865) > > > > However in our data model we don't have a single field which we can > > filter by, it's more a case of getting a list of all the IDs of > > documents the user can access and then searching them. > > > > Eg, modelled as a boolean query, which works but isn't very elegant > > and I can see it having issues with performance as the query grows: > > > > ``` > > { > > :query => { > > :bool => { > > :must => [ > > { > > :query_string => { > > :query => params[:q] > > } > > }, > > { > > :query_string => { > > :default_field => "id", > > :query => accessible_ids.join(" OR ") > > } > > } > > ] > > } > > } > > } > > > > ``` > > > > Is there a way of performing a term query against an array of terms so > > that I could convert this to a filter? > > Or is there a better way of doing this that someone knows of? > > > > Thanks in advance. > > > > -- > > Richard Livsey > > Minutebase - Online Meeting Minutes > > [http://minutebase.com](http://minutebase.com) > > [http://livsey.org](http://livsey.org) -- Richard Livsey Minutebase - Online Meeting Minutes [http://minutebase.com](http://minutebase.com) > **[Livsey.org](http://livsey.org)** > > I’ve been using Firebase with Ember.js quite a lot recently and have just > released Fireplace, a library to integrate the two more easily. It ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 24, 2010, 6:05pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/5 "2010-08-24T18:05:02Z")
The good news is that if you use filter (either constant\_score wrapping a query or terms filter) is that the same effectiveness of your caching of doc ids, so you should get good performance, even with a large list. Regarding the list below, you can index that information on the document level. For example, add created\_by, shared\_with, status (public or not), and so on. Then, you can use an "or" filter on them. Thats possible assuming that those "..." don't hide many more complex restrictions 😉 -shay.banon On Tue, Aug 24, 2010 at 8:59 PM, Richard Livsey [richard@livsey.org](mailto:richard@livsey.org) wrote: > I wish it was simpler, but there's no real grouping of documents on a > per-user basis. > A user can see a document if any of the following are true (and a few > other conditions): > > - they are an admin > - they created the document > - they are a member of the project the document belongs to > - the document is "shared" with them > - the document is public > - etc... > > The query to generate this list of documents can be fairly hairy, so > we cache a list of document IDs on a per-user basis. > So when doing a full-text search I have the IDs to restrict against, > but unfortunately it could number in the hundreds. > > Thanks. > > On Tue, Aug 24, 2010 at 6:53 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > There is the terms filter which should fit better in this case. Can you > > explain more why you get all the ids and then do a search? How do you get > > all the ids? Can't that be "logical" groups that are added to each doc, > > and > > filter by that? It will be much more efficient. > > -shay.banon > > > > On Tue, Aug 24, 2010 at 8:43 PM, Richard Livsey [livsey@gmail.com](mailto:livsey@gmail.com) > > wrote: > > > > > Each user in our app can see a subset of the documents based on a > > > number of conditions (permissions etc...) and I'm wondering what the > > > best approach to building a restricted query would be. > > > > > > Andrei asked a similar question about restricting the query with > > > boolean vs filters in which filters came up as the most efficient > > > method: > > [http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865](http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filters-and-more-td1177230.html;cid=1282670994600-452#a1182865) > > > > However in our data model we don't have a single field which we can > > > filter by, it's more a case of getting a list of all the IDs of > > > documents the user can access and then searching them. > > > > > > Eg, modelled as a boolean query, which works but isn't very elegant > > > and I can see it having issues with performance as the query grows: > > > > > > ``` > > > { > > > :query => { > > > :bool => { > > > :must => [ > > > { > > > :query_string => { > > > :query => params[:q] > > > } > > > }, > > > { > > > :query_string => { > > > :default_field => "id", > > > :query => accessible_ids.join(" OR ") > > > } > > > } > > > ] > > > } > > > } > > > } > > > > > > ``` > > > > > > Is there a way of performing a term query against an array of terms so > > > that I could convert this to a filter? > > > Or is there a better way of doing this that someone knows of? > > > > > > Thanks in advance. > > > > > > -- > > > Richard Livsey > > > Minutebase - Online Meeting Minutes > > > [http://minutebase.com](http://minutebase.com) > > > [http://livsey.org](http://livsey.org) > > -- > Richard Livsey > Minutebase - Online Meeting Minutes > [http://minutebase.com](http://minutebase.com) > [http://livsey.org](http://livsey.org) ---
**Author:** ![Richard\_Livsey\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richard_livsey_2/32/3241_2.png) [@Richard\_Livsey\_2](https://discuss.elastic.co/u/Richard_Livsey_2)\ **Post date:** [August 24, 2010, 6:14pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/6 "2010-08-24T18:14:23Z")
Ah, that looks along the lines of what I'm trying to do, thanks. I had tried something similar previously but was confused by the note in the docs saying: "The filter object can hold only filter elements, not queries", so had given up trying to get it working as part of the filter! What does that note actually mean? The docs also mention that filters don't perform any scoring, so is there a benefit of nesting the query in a constant\_score in your example? I've managed to get the following working now: { "query" =\> { "filtered" =\> { "query" =\> { "query\_string" =\> { "query" =\> "test" } }, "filter" =\> { "query" =\> { "query\_string" =\> { "default\_field" =\> "document\_id", "query" =\> "4c31d38acf02a31548000009 OR 4c31d38acf02a31548000008" } } } } } } Thanks for the help! On Tue, Aug 24, 2010 at 6:56 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Hi Richard, > I have a similar situation where for any user we construct a query > string that enforces what they are entitled to. This query string can > be arbitrarily complex. Just taking that query string and wrapping it > in a filter seems to work very well. So, my final query typically ends > up looking like: > > ``` > 'filtered' : { > 'query' : { > 'query_string' : {'query':mainquery} > }, > 'constant_score' : { > 'filter' : { > 'query' : { > 'query_string' : {'query':userentitlements} > } > } > } > } > > ``` > > So you can keep doing similar to what you are doing, but this query > structure yields good caching on the user entitlements part. > > Not sure if this is optimal, but is working quite well in my > performance testing thus far. > > Regards, > Paul > > On Aug 24, 11:43 am, Richard Livsey [liv...@gmail.com](mailto:liv...@gmail.com) wrote: > > > Each user in our app can see a subset of the documents based on a > > number of conditions (permissions etc...) and I'm wondering what the > > best approach to building a restricted query would be. > > > > Andrei asked a similar question about restricting the query with > > boolean vs filters in which filters came up as the most efficient > > method:[http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filt](http://elasticsearch-users.115913.n3.nabble.com/Boolean-query-vs-filt)... > > > > However in our data model we don't have a single field which we can > > filter by, it's more a case of getting a list of all the IDs of > > documents the user can access and then searching them. > > > > Eg, modelled as a boolean query, which works but isn't very elegant > > and I can see it having issues with performance as the query grows: > > > > ``` > > { > > :query => { > > :bool => { > > :must => [ > > { > > :query_string => { > > :query => params[:q] > > } > > }, > > { > > :query_string => { > > :default_field => "id", > > :query => accessible_ids.join(" OR ") > > } > > } > > ] > > } > > } > > } > > > > ``` > > > > Is there a way of performing a term query against an array of terms so > > that I could convert this to a filter? > > Or is there a better way of doing this that someone knows of? ---
**Author:** ![Richard\_Livsey\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richard_livsey_2/32/3241_2.png) [@Richard\_Livsey\_2](https://discuss.elastic.co/u/Richard_Livsey_2)\ **Post date:** [August 24, 2010, 6:17pm UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/7 "2010-08-24T18:17:44Z")
Fantastic, thanks for the help. The actual case is a bit more complicated, the documents themselves aren't the items which have the associated created\_by/shared\_with data on them. They are children of those items, so every time one of the parents changed I'd have to update the same data on all the children. As one parent object can have many hundreds of child objects, that would get to be a huge amount of updates. It sounds like passing in the list of id's isn't _too_ crazy, so I'll carry on down these lines for now and do some performance testing against a decent sized dataset. Cheers again for the help! On Tue, Aug 24, 2010 at 7:05 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > The good news is that if you use filter (either constant\_score wrapping a > query or terms filter) is that the same effectiveness of your caching of doc > ids, so you should get good performance, even with a large list. > Regarding the list below, you can index that information on the document > level. For example, add created\_by, shared\_with, status (public or not), and > so on. Then, you can use an "or" filter on them. Thats possible assuming > that those "..." don't hide many more complex restrictions 😉 > -shay.banon > On Tue, Aug 24, 2010 at 8:59 PM, Richard Livsey [richard@livsey.org](mailto:richard@livsey.org) wrote: > > > I wish it was simpler, but there's no real grouping of documents on a > > per-user basis. > > A user can see a document if any of the following are true (and a few > > other conditions): > > > > - they are an admin > > - they created the document > > - they are a member of the project the document belongs to > > - the document is "shared" with them > > - the document is public > > - etc... > > > > The query to generate this list of documents can be fairly hairy, so > > we cache a list of document IDs on a per-user basis. > > So when doing a full-text search I have the IDs to restrict against, > > but unfortunately it could number in the hundreds. > > > > Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/match-array-of-terms-in-filtered-query/3250/8 "2017-07-06T04:20:23Z")