# Two-tiered filtering of results
**URL:** https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437
**Category:** Elasticsearch
**Created:** [April 3, 2013, 9:58pm UTC](https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437 "2013-04-03T21:58:23Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@James\_Harrison](https://discuss.elastic.co/u/James_Harrison)
#### Post date: [April 3, 2013, 9:58pm UTC](https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437/1 "2013-04-03T21:58:23Z")
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
I'm working on a site which has a decent number of tagged documents
(~2m). ES indexes the tags (plus a bunch of other fields) and users
navigate the site using pages with size/from, nothing very fancy
there. ES is just storing the documents; everything else lives in
MongoDB (including the docs themselves, we just use ES for querying
and display - it's a Rails app using tire as an ES adapter).
We allow users to hide (completely remove) and obscure (they won't see
unless they click through) results by storing lists of tags on user
objects, and doing a one-by-one check (are any of $document's tags in
$user's obscured list, if so obscure, if not render normally) for
obscuring and a not terms filter for hiding documents.
We're now looking at letting users use complex hiding/obscuring -
using booleans, mainly - by storing queries (ie 'tag1 OR (tag2 AND
tag3)') on the user object and searching using that as a filter
instead of our current not terms filter, but when it comes to
obscuring we're having some issues working out how to map that to ES
queries.
Ideally we'd get the hidden-filtered, obscured-annotated list back
from ES somehow but I don't think (having read the docs) that it's
feasible to do this.
Am I wrong in thinking I'm going to have to make a second query with
the results from the first to see which docs match the obscured
documents query, then marry the results up in the application, or is
there a neater solution?
* * *
Cheers,
James Harrison
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iEYEARECAAYFAlFcpf8ACgkQ22kkGnnJQAzXLwCgp5MfClOpRPwbmv/dajpkiMDf
S3kAn18629NZXH3MLkSYzdN6B52iXUXe
=vd3i
-----END PGP SIGNATURE-----
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [April 5, 2013, 2:04pm UTC](https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437/2 "2013-04-05T14:04:32Z")
How do you sort your documents?
On Wednesday, April 3, 2013 5:58:23 PM UTC-4, James Harrison wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> I'm working on a site which has a decent number of tagged documents
> (~2m). ES indexes the tags (plus a bunch of other fields) and users
> navigate the site using pages with size/from, nothing very fancy
> there. ES is just storing the documents; everything else lives in
> MongoDB (including the docs themselves, we just use ES for querying
> and display - it's a Rails app using tire as an ES adapter).
>
> We allow users to hide (completely remove) and obscure (they won't see
> unless they click through) results by storing lists of tags on user
> objects, and doing a one-by-one check (are any of $document's tags in
> $user's obscured list, if so obscure, if not render normally) for
> obscuring and a not terms filter for hiding documents.
>
> We're now looking at letting users use complex hiding/obscuring -
> using booleans, mainly - by storing queries (ie 'tag1 OR (tag2 AND
> tag3)') on the user object and searching using that as a filter
> instead of our current not terms filter, but when it comes to
> obscuring we're having some issues working out how to map that to ES
> queries.
>
> Ideally we'd get the hidden-filtered, obscured-annotated list back
> from ES somehow but I don't think (having read the docs) that it's
> feasible to do this.
>
> Am I wrong in thinking I'm going to have to make a second query with
> the results from the first to see which docs match the obscured
> documents query, then marry the results up in the application, or is
> there a neater solution?
>
> * * *
>
> Cheers,
> James Harrison
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
>
> iEYEARECAAYFAlFcpf8ACgkQ22kkGnnJQAzXLwCgp5MfClOpRPwbmv/dajpkiMDf
> S3kAn18629NZXH3MLkSYzdN6B52iXUXe
> =vd3i
> -----END PGP SIGNATURE-----
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@James\_Harrison](https://discuss.elastic.co/u/James_Harrison)
#### Post date: [April 6, 2013, 10:51am UTC](https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437/3 "2013-04-06T10:51:58Z")
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Just a simple sort on an incrementing ID field (or another static
field), never score.
Cheers,
James Harrison
On 05/04/2013 15:04, Igor Motov wrote:
> How do you sort your documents?
>
> On Wednesday, April 3, 2013 5:58:23 PM UTC-4, James Harrison
> wrote:
>
> Hi all,
>
> I'm working on a site which has a decent number of tagged
> documents (~2m). ES indexes the tags (plus a bunch of other fields)
> and users navigate the site using pages with size/from, nothing
> very fancy there. ES is just storing the documents; everything else
> lives in MongoDB (including the docs themselves, we just use ES for
> querying and display - it's a Rails app using tire as an ES
> adapter).
>
> We allow users to hide (completely remove) and obscure (they won't
> see unless they click through) results by storing lists of tags on
> user objects, and doing a one-by-one check (are any of $document's
> tags in $user's obscured list, if so obscure, if not render
> normally) for obscuring and a not terms filter for hiding
> documents.
>
> We're now looking at letting users use complex hiding/obscuring -
> using booleans, mainly - by storing queries (ie 'tag1 OR (tag2 AND
> tag3)') on the user object and searching using that as a filter
> instead of our current not terms filter, but when it comes to
> obscuring we're having some issues working out how to map that to
> ES queries.
>
> Ideally we'd get the hidden-filtered, obscured-annotated list back
> from ES somehow but I don't think (having read the docs) that it's
> feasible to do this.
>
> Am I wrong in thinking I'm going to have to make a second query
> with the results from the first to see which docs match the
> obscured documents query, then marry the results up in the
> application, or is there a neater solution?
>
> -- You received this message because you are subscribed to the
> Google Groups "elasticsearch" group. To unsubscribe from this group
> and stop receiving emails from it, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit
> [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iEYEARECAAYFAlFf/k4ACgkQ22kkGnnJQAwwcwCeLdx+Ic1Genl/ed+Np5DAFcgY
oSAAmQHa6OPFoAr4w/gPMIDz7f99qEfL
=/umm
-----END PGP SIGNATURE-----
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [April 6, 2013, 1:20pm UTC](https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437/4 "2013-04-06T13:20:27Z")
If you are not using score to sort, you can use it to figure out which
records match a certain filter. All you need to do is wrap your query into
constant\_score\_query[http://www.elasticsearch.org/guide/reference/query-dsl/constant-score-query/](http://www.elasticsearch.org/guide/reference/query-dsl/constant-score-query/) to
ensure that your query by itself returns the same \_score for all records.
Then you can wrap this constant\_score\_query into custom\_filters\_score[http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query/](http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query/)query, express your show/hide logic as one of the filters with some boost
(let's use 2, for example) and set "track\_scores": true. This way all
records that match your filter will have score of 2.0 and all recored that
don't match your filter will have score of 1.0.
On Saturday, April 6, 2013 6:51:58 AM UTC-4, James Harrison wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Just a simple sort on an incrementing ID field (or another static
> field), never score.
>
> Cheers,
> James Harrison
>
> On 05/04/2013 15:04, Igor Motov wrote:
>
> > How do you sort your documents?
> >
> > On Wednesday, April 3, 2013 5:58:23 PM UTC-4, James Harrison
> > wrote:
> >
> > Hi all,
> >
> > I'm working on a site which has a decent number of tagged
> > documents (~2m). ES indexes the tags (plus a bunch of other fields)
> > and users navigate the site using pages with size/from, nothing
> > very fancy there. ES is just storing the documents; everything else
> > lives in MongoDB (including the docs themselves, we just use ES for
> > querying and display - it's a Rails app using tire as an ES
> > adapter).
> >
> > We allow users to hide (completely remove) and obscure (they won't
> > see unless they click through) results by storing lists of tags on
> > user objects, and doing a one-by-one check (are any of $document's
> > tags in $user's obscured list, if so obscure, if not render
> > normally) for obscuring and a not terms filter for hiding
> > documents.
> >
> > We're now looking at letting users use complex hiding/obscuring -
> > using booleans, mainly - by storing queries (ie 'tag1 OR (tag2 AND
> > tag3)') on the user object and searching using that as a filter
> > instead of our current not terms filter, but when it comes to
> > obscuring we're having some issues working out how to map that to
> > ES queries.
> >
> > Ideally we'd get the hidden-filtered, obscured-annotated list back
> > from ES somehow but I don't think (having read the docs) that it's
> > feasible to do this.
> >
> > Am I wrong in thinking I'm going to have to make a second query
> > with the results from the first to see which docs match the
> > obscured documents query, then marry the results up in the
> > application, or is there a neater solution?
> >
> > -- You received this message because you are subscribed to the
> > Google Groups "elasticsearch" group. To unsubscribe from this group
> > and stop receiving emails from it, send an email to
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. For more options, visit
> > [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
>
> iEYEARECAAYFAlFf/k4ACgkQ22kkGnnJQAwwcwCeLdx+Ic1Genl/ed+Np5DAFcgY
> oSAAmQHa6OPFoAr4w/gPMIDz7f99qEfL
> =/umm
> -----END PGP SIGNATURE-----
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@James\_Harrison](https://discuss.elastic.co/u/James_Harrison)
#### Post date: [April 7, 2013, 1:45am UTC](https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437/5 "2013-04-07T01:45:20Z")
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
That sounds very workable as an approach - I'll give it a shot. Thanks.
Cheers,
James Harrison
On 06/04/2013 14:20, Igor Motov wrote:
> If you are not using score to sort, you can use it to figure out
> which records match a certain filter. All you need to do is wrap
> your query into constant\_score\_query
> [http://www.elasticsearch.org/guide/reference/query-dsl/constant-score-query/](http://www.elasticsearch.org/guide/reference/query-dsl/constant-score-query/)
> to ensure that your query by itself returns the same \_score for
> all records. Then you can wrap this constant\_score\_query into
> custom\_filters\_score
> [http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query/](http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query/)
query, express your show/hide logic as one of the filters with some
> boost (let's use 2, for example) and set "track\_scores": true. This
> way all records that match your filter will have score of 2.0 and
> all recored that don't match your filter will have score of 1.0.
>
> On Saturday, April 6, 2013 6:51:58 AM UTC-4, James Harrison wrote:
>
> Just a simple sort on an incrementing ID field (or another static
> field), never score.
>
> Cheers, James Harrison
>
> On 05/04/2013 15:04, Igor Motov wrote:
>
> > How do you sort your documents?
>
> > On Wednesday, April 3, 2013 5:58:23 PM UTC-4, James Harrison
> > wrote:
>
> > Hi all,
>
> > I'm working on a site which has a decent number of tagged
> > documents (~2m). ES indexes the tags (plus a bunch of other
> > fields) and users navigate the site using pages with size/from,
> > nothing very fancy there. ES is just storing the documents;
> > everything else lives in MongoDB (including the docs themselves,
> > we just use ES for querying and display - it's a Rails app using
> > tire as an ES adapter).
>
> > We allow users to hide (completely remove) and obscure (they
> > won't see unless they click through) results by storing lists of
> > tags on user objects, and doing a one-by-one check (are any of
> > $document's tags in $user's obscured list, if so obscure, if not
> > render normally) for obscuring and a not terms filter for hiding
> > documents.
>
> > We're now looking at letting users use complex hiding/obscuring
> >
> > - using booleans, mainly - by storing queries (ie 'tag1 OR (tag2
> > AND tag3)') on the user object and searching using that as a
> > filter instead of our current not terms filter, but when it comes
> > to obscuring we're having some issues working out how to map that
> > to ES queries.
>
> > Ideally we'd get the hidden-filtered, obscured-annotated list
> > back from ES somehow but I don't think (having read the docs)
> > that it's feasible to do this.
>
> > Am I wrong in thinking I'm going to have to make a second query
> > with the results from the first to see which docs match the
> > obscured documents query, then marry the results up in the
> > application, or is there a neater solution?
>
> > -- You received this message because you are subscribed to the
> > Google Groups "elasticsearch" group. To unsubscribe from this
> > group and stop receiving emails from it, send an email to
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. For more
> > options,
> > visit
> > [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> -- You received this message because you are subscribed to the
> Google Groups "elasticsearch" group. To unsubscribe from this group
> and stop receiving emails from it, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit
> [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iEYEARECAAYFAlFgz7AACgkQ22kkGnnJQAwTlACZAcYkyAJ+6/S8aBII3oTUkGev
XFsAn3pduR9LhGtilXVjD/zNhfAHwnyx
=/mTG
-----END PGP SIGNATURE-----
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:42am UTC](https://discuss.elastic.co/t/two-tiered-filtering-of-results/11437/6 "2017-07-06T02:42:26Z")