# Unable to filter out results based on 'missing' fields

**URL:** <https://discuss.elastic.co/t/unable-to-filter-out-results-based-on-missing-fields/7977>\
**Category:** Elasticsearch\
**Created:** [June 5, 2012, 7:21am UTC](https://discuss.elastic.co/t/unable-to-filter-out-results-based-on-missing-fields/7977 "2012-06-05T07:21:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![aash\_dhariya](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@aash\_dhariya](https://discuss.elastic.co/u/aash_dhariya)\
**Post date:** [June 5, 2012, 7:21am UTC](https://discuss.elastic.co/t/unable-to-filter-out-results-based-on-missing-fields/7977/1 "2012-06-05T07:21:23Z")

</div>

I am using tire-0.4.2 to interact with elasticsearch in my rails  
application (uses mongodb database and mongoid for interacting with  
mongdb). I have a post model which has a spam document embedded in it.  
_post.rb_

include Mongoid::Document  
include Mongoid::Timestamps\* \*  
..  
embeds one :spam, as: :spammable  
...

_spam.rb_  
include Mongoid::Document  
include Mongoid::Timestamps  
embedded\_in :spammable, polymorphic: true  
field :needs\_approval, type: Boolean, default: false  
field :is\_spam, type: Time

has\_and\_belongs\_to\_many :request\_spam\_by, :class\_name =\> "User"

field :request\_spam, type: Boolean, default: false

I want to get all the posts which have no spam document: Here is the tire  
query

Post.tire.search(:load =\> :true, page: self.page, per\_page: Post::PER\_PAGE)  
do |pf|  
pf.query{ |query| query.string self.search\_text } unless  
search\_text.blank?  
pf.filter(:missing, :field =\> 'spam')  
pf.filter(:term, :college\_id =\> self.college.id)  
pf.filter(:term, :user\_id =\> self.user.id)  
pf.filter(:missing, :field =\> 'spam' )  
pf.filter(:terms, :user\_type =\> self.user\_type) unless  
self.user\_type.blank?  
pf.filter(:range, :created\_at =\> {:gte =\> self.from\_time}) unless  
self.from\_time.blank?  
pf.filter(:range, :created\_at =\> {:lte =\> self.to\_time}) unless  
self.to\_time.blank?  
pf.sort{|s| s.by :updated\_at, self.sort\_order}  
end

elasticsearch query:

curl -X GET  
"[http://localhost:9200/development\_posts/post/\_search?from=0&load=true&page=1&per\_page=10&size=10&pretty=true](http://localhost:9200/development_posts/post/_search?from=0&load=true&page=1&per_page=10&size=10&pretty=true)"  
-d  
'{"sort":[{"updated\_at":"desc"}],"filter":{"and":[{"missing":{"field":"spam"}},{"term":{"college\_id":"4fb424a5addf32296f00013a"}},  
{"missing":{"field":"spam"}},{"range":{"created\_at":{"gte":"2012-06-05T00:00:00+05:30"}}},{"range":{"created\_at":{"lte":"2012-06-05T23:59:59+05:30"}}}]},"size":10,"from":0}'

The results of the query gives me the documents in which spam exists even  
though I am search only for document which have spam document missing. I  
don't know what is the mistake I am doing. Can anybody point me in the  
right direction?

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 8, 2012, 10:17pm UTC](https://discuss.elastic.co/t/unable-to-filter-out-results-based-on-missing-fields/7977/2 "2012-06-08T22:17:38Z")

</div>

Your search seems fine, can you gist a sample that curls some docs and  
shows the behavior? (simpler to understand whats going on)? Also, for  
performance reasons, its better not to use the "filter" search element  
unless you use facets and instead use constant\_score wrapping that filter,  
or filtered query (under the "query" element).

On Tue, Jun 5, 2012 at 9:21 AM, geeky\_sh [aash.discover@gmail.com](mailto:aash.discover@gmail.com) wrote:

> I am using tire-0.4.2 to interact with elasticsearch in my rails  
> application (uses mongodb database and mongoid for interacting with  
> mongdb). I have a post model which has a spam document embedded in it.  
> _post.rb_
> 
> include Mongoid::Document  
> include Mongoid::Timestamps\* \*  
> ..  
> embeds one :spam, as: :spammable  
> ...
> 
> _spam.rb_  
> include Mongoid::Document  
> include Mongoid::Timestamps  
> embedded\_in :spammable, polymorphic: true  
> field :needs\_approval, type: Boolean, default: false  
> field :is\_spam, type: Time
> 
> has\_and\_belongs\_to\_many :request\_spam\_by, :class\_name =\> "User"
> 
> field :request\_spam, type: Boolean, default: false
> 
> I want to get all the posts which have no spam document: Here is the tire  
> query
> 
> Post.tire.search(:load =\> :true, page: self.page, per\_page:  
> Post::PER\_PAGE) do |pf|  
> pf.query{ |query| query.string self.search\_text } unless  
> search\_text.blank?  
> pf.filter(:missing, :field =\> 'spam')  
> pf.filter(:term, :college\_id =\> self.college.id)  
> pf.filter(:term, :user\_id =\> self.user.id)  
> pf.filter(:missing, :field =\> 'spam' )  
> pf.filter(:terms, :user\_type =\> self.user\_type) unless  
> self.user\_type.blank?  
> pf.filter(:range, :created\_at =\> {:gte =\> self.from\_time}) unless  
> self.from\_time.blank?  
> pf.filter(:range, :created\_at =\> {:lte =\> self.to\_time}) unless  
> self.to\_time.blank?  
> pf.sort{|s| s.by :updated\_at, self.sort\_order}  
> end
> 
> elasticsearch query:
> 
> curl -X GET "  
> [http://localhost:9200/development\_posts/post/\_search?from=0&load=true&page=1&per\_page=10&size=10&pretty=true](http://localhost:9200/development_posts/post/_search?from=0&load=true&page=1&per_page=10&size=10&pretty=true)"  
> -d
> 
> '{"sort":[{"updated\_at":"desc"}],"filter":{"and":[{"missing":{"field":"spam"}},{"term":{"college\_id":"4fb424a5addf32296f00013a"}},
> 
> {"missing":{"field":"spam"}},{"range":{"created\_at":{"gte":"2012-06-05T00:00:00+05:30"}}},{"range":{"created\_at":{"lte":"2012-06-05T23:59:59+05:30"}}}]},"size":10,"from":0}'
> 
> The results of the query gives me the documents in which spam exists even  
> though I am search only for document which have spam document missing. I  
> don't know what is the mistake I am doing. Can anybody point me in the  
> right direction?

---

<div class="post-metadata">

**Author:** ![ywtsang](https://avatars.discourse-cdn.com/v4/letter/y/a698b9/32.png) [@ywtsang](https://discuss.elastic.co/u/ywtsang)\
**Post date:** [June 9, 2012, 7:37am UTC](https://discuss.elastic.co/t/unable-to-filter-out-results-based-on-missing-fields/7977/3 "2012-06-09T07:37:09Z")

</div>

what do you mean about the performance reason?

to my understanding, filter does not affect scoring and can be easily  
cached, therefore i use filter instead of query

On Jun 9, 2012 6:17 AM, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> Your search seems fine, can you gist a sample that curls some docs and  
> shows the behavior? (simpler to understand whats going on)? Also, for  
> performance reasons, its better not to use the "filter" search element  
> unless you use facets and instead use constant\_score wrapping that filter,  
> or filtered query (under the "query" element).
> 
> On Tue, Jun 5, 2012 at 9:21 AM, geeky\_sh [aash.discover@gmail.com](mailto:aash.discover@gmail.com) wrote:
> 
> > I am using tire-0.4.2 to interact with elasticsearch in my rails  
> > application (uses mongodb database and mongoid for interacting with  
> > mongdb). I have a post model which has a spam document embedded in it.  
> > post.rb
> > 
> > include Mongoid::Document  
> > include Mongoid::Timestamps  
> > ..  
> > embeds one :spam, as: :spammable  
> > ...
> > 
> > spam.rb  
> > include Mongoid::Document  
> > include Mongoid::Timestamps  
> > embedded\_in :spammable, polymorphic: true  
> > field :needs\_approval, type: Boolean, default: false  
> > field :is\_spam, type: Time
> > 
> > has\_and\_belongs\_to\_many :request\_spam\_by, :class\_name =\> "User"
> > 
> > field :request\_spam, type: Boolean, default: false
> > 
> > I want to get all the posts which have no spam document: Here is the  
> > tire query
> > 
> > Post.tire.search(:load =\> :true, page: self.page, per\_page:  
> > Post::PER\_PAGE) do |pf|  
> > pf.query{ |query| query.string self.search\_text } unless  
> > search\_text.blank?  
> > pf.filter(:missing, :field =\> 'spam')  
> > pf.filter(:term, :college\_id =\> self.college.id)  
> > pf.filter(:term, :user\_id =\> self.user.id)  
> > pf.filter(:missing, :field =\> 'spam' )  
> > pf.filter(:terms, :user\_type =\> self.user\_type) unless  
> > self.user\_type.blank?  
> > pf.filter(:range, :created\_at =\> {:gte =\> self.from\_time})  
> > unless self.from\_time.blank?  
> > pf.filter(:range, :created\_at =\> {:lte =\> self.to\_time}) unless  
> > self.to\_time.blank?  
> > pf.sort{|s| s.by :updated\_at, self.sort\_order}  
> > end
> > 
> > elasticsearch query:
> > 
> > curl -X GET "  
> > [http://localhost:9200/development\_posts/post/\_search?from=0&load=true&page=1&per\_page=10&size=10&pretty=true](http://localhost:9200/development_posts/post/_search?from=0&load=true&page=1&per_page=10&size=10&pretty=true)"  
> > -d

'{"sort":[{"updated\_at":"desc"}],"filter":{"and":[{"missing":{"field":"spam"}},{"term":{"college\_id":"4fb424a5addf32296f00013a"}},

> >

{"missing":{"field":"spam"}},{"range":{"created\_at":{"gte":"2012-06-05T00:00:00+05:30"}}},{"range":{"created\_at":{"lte":"2012-06-05T23:59:59+05:30"}}}]},"size":10,"from":0}'

> > The results of the query gives me the documents in which spam exists  
> > even though I am search only for document which have spam document missing.  
> > I don't know what is the mistake I am doing. Can anybody point me in the  
> > right direction?

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [June 9, 2012, 9:00am UTC](https://discuss.elastic.co/t/unable-to-filter-out-results-based-on-missing-fields/7977/4 "2012-06-09T09:00:02Z")

</div>

On Sat, 2012-06-09 at 15:37 +0800, Yiu Wing TSANG wrote:

> what do you mean about the performance reason?
> 
> to my understanding, filter does not affect scoring and can be easily  
> cached, therefore i use filter instead of query

You can filter a query, or you can filter results AFTER the query has  
been run and the facets have been calculated on the original results.

You are using the second type of filter but you're not using facets, so  
you are running your query, then applying a filter:  
{  
query: { query\_string: { query: "string"}},  
filter: { ...your filter... }  
}

Much better in this case to use a filtered query, or a constant score  
query, for instance:

{  
query: {  
filtered: {  
query: { query\_string: { query: "string"}},  
filter: { ...your filter... }  
}  
}  
}

clint

> On Jun 9, 2012 6:17 AM, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> 
> > Your search seems fine, can you gist a sample that curls some docs  
> > and shows the behavior? (simpler to understand whats going on)? Also,  
> > for performance reasons, its better not to use the "filter" search  
> > element unless you use facets and instead use constant\_score wrapping  
> > that filter, or filtered query (under the "query" element).
> > 
> > On Tue, Jun 5, 2012 at 9:21 AM, geeky\_sh [aash.discover@gmail.com](mailto:aash.discover@gmail.com)  
> > wrote:
> > 
> > > I am using tire-0.4.2 to interact with elasticsearch in my rails  
> > > application (uses mongodb database and mongoid for interacting with  
> > > mongdb). I have a post model which has a spam document embedded in it.  
> > > post.rb
> > > 
> > > include Mongoid::Document  
> > > include Mongoid::Timestamps  
> > > ..  
> > > embeds one :spam, as: :spammable  
> > > ...
> > > 
> > > spam.rb  
> > > include Mongoid::Document  
> > > include Mongoid::Timestamps  
> > > embedded\_in :spammable, polymorphic: true  
> > > field :needs\_approval, type: Boolean, default: false  
> > > field :is\_spam, type: Time
> > > 
> > > has\_and\_belongs\_to\_many :request\_spam\_by, :class\_name =\> "User"
> > > 
> > > field :request\_spam, type: Boolean, default: false
> > > 
> > > I want to get all the posts which have no spam document: Here is  
> > > the tire query
> > > 
> > > Post.tire.search(:load =\> :true, page: self.page, per\_page:  
> > > Post::PER\_PAGE) do |pf|  
> > > pf.query{ |query| query.string self.search\_text } unless  
> > > search\_text.blank?  
> > > pf.filter(:missing, :field =\> 'spam')  
> > > pf.filter(:term, :college\_id =\> self.college.id)  
> > > pf.filter(:term, :user\_id =\> self.user.id)  
> > > pf.filter(:missing, :field =\> 'spam' )  
> > > pf.filter(:terms, :user\_type =\> self.user\_type) unless  
> > > self.user\_type.blank?  
> > > pf.filter(:range, :created\_at =\> {:gte =\> self.from\_time})  
> > > unless self.from\_time.blank?  
> > > pf.filter(:range, :created\_at =\> {:lte =\> self.to\_time})  
> > > unless self.to\_time.blank?  
> > > pf.sort{|s| s.by :updated\_at, self.sort\_order}  
> > > end
> > > 
> > > elasticsearch query:
> > > 
> > > curl -X GET  
> > > "[http://localhost:9200/development\_posts/post/\_search?from=0&load=true&page=1&per\_page=10&size=10&pretty=true](http://localhost:9200/development_posts/post/_search?from=0&load=true&page=1&per_page=10&size=10&pretty=true)" -d
> 
> '{"sort":[{"updated\_at":"desc"}],"filter":{"and":[{"missing":{"field":"spam"}},{"term":{"college\_id":"4fb424a5addf32296f00013a"}},
> 
> > >
> 
> {"missing":{"field":"spam"}},{"range":{"created\_at":{"gte":"2012-06-05T00:00:00+05:30"}}},{"range":{"created\_at":{"lte":"2012-06-05T23:59:59+05:30"}}}]},"size":10,"from":0}'
> 
> > > The results of the query gives me the documents in which spam  
> > > exists even though I am search only for document which have spam  
> > > document missing. I don't know what is the mistake I am doing. Can  
> > > anybody point me in the right direction?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:25am UTC](https://discuss.elastic.co/t/unable-to-filter-out-results-based-on-missing-fields/7977/5 "2017-07-06T03:25:06Z")

</div>