# Query for missing fields using a score

**URL:** <https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303>\
**Category:** Elasticsearch\
**Created:** [October 10, 2012, 10:32am UTC](https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303 "2012-10-10T10:32:38Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amy](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Amy](https://discuss.elastic.co/u/Amy)\
**Post date:** [October 10, 2012, 10:32am UTC](https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303/1 "2012-10-10T10:32:38Z")

</div>

Hi,  
I had a query to search for certain fields which was scored using a custom  
script. Then I needed to take account of missing data and found that I'd  
need to use a filter for this. The solution query (simplified) is:  
{  
"query": {  
"custom\_filters\_score": {  
"query": {  
"match\_all": {}  
},  
"filters": [  
{  
"script": "((doc['id'].value == 'hi') ? 2 : 0) + 20",  
"filter": {  
"bool": {  
"should": [  
{  
"term": {  
"id": "hello"  
}  
},  
{  
"term": {  
"id": "hi"  
}  
},  
{  
"missing": {  
"field": "jurisdiction",  
"existence": true,  
"null\_value": true  
}  
}  
]  
}  
}  
}  
]  
}  
},  
"size": 30  
}

My problem is that the 'match\_all' query _will return all the data_,  
whereas I only want data matched by my filter to be returned. By doctoring  
the scoring script I can ensure that filtered data is top of the list, but  
I'm worried about performance.

My original query returned only data matching my terms, but I was unable  
to use missing data with that. The original query is:  
{  
"query": {  
"custom\_score": {  
"query": {  
"bool": {  
"should": [  
{  
"term": {  
"id": "hi"  
}  
},  
{  
"term": {  
"id": "hello"  
}  
}  
]  
}  
},  
"script": "((doc['id'].value == 'hi') ? 2 : 0)"  
}  
}  
}

Please note that this is a simplified query, I need the script rather than  
using a boost.

Is there a better solution?

--

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [October 11, 2012, 8:10am UTC](https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303/2 "2012-10-11T08:10:23Z")

</div>

Hi Amy

> I had a query to search for certain fields which was scored using a  
> custom script. Then I needed to take account of missing data and found  
> that I'd need to use a filter for this. The solution query  
> (simplified) is:

> My problem is that the 'match\_all' query will return all the data,  
> whereas I only want data matched by my filter to be returned. By  
> doctoring the scoring script I can ensure that filtered data is top of  
> the list, but I'm worried about performance.
> 
> My original query returned only data matching my terms, but I was  
> unable to use missing data with that. The original query is:

I'm unclear from your examples exactly what your query should do. Could  
you explain in english what you want to achieve?

ta

clint

> {  
> "query": {  
> "custom\_score": {  
> "query": {  
> "bool": {  
> "should": [  
> {  
> "term": {  
> "id": "hi"  
> }  
> },  
> {  
> "term": {  
> "id": "hello"  
> }  
> }  
> ]  
> }  
> },  
> "script": "((doc['id'].value == 'hi') ? 2 : 0)"  
> }  
> }  
> }
> 
> Please note that this is a simplified query, I need the script rather  
> than using a boost.
> 
> Is there a better solution?
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Amy](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Amy](https://discuss.elastic.co/u/Amy)\
**Post date:** [October 11, 2012, 12:58pm UTC](https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303/3 "2012-10-11T12:58:40Z")

</div>

Hi,  
To clarify, in this example I want to return all documents where the term  
"id" is either "hi" or "hello" or is missing altogether.  
In the query I provided (without the filter) documents without the term  
"id" were not returned, which was not what I wanted.  
In the query that had the filter all documents were returned, which was  
also not what I wanted, but I was able to bring the documents I wanted to  
the top of the search by adding 20 to the score in the filter. This is not  
ideal, and I'm worried about performance, even if I were to change the size  
returned to 1.  
Does that explain my predicament?

On Thursday, 11 October 2012 09:10:36 UTC+1, Clinton Gormley wrote:

> Hi Amy
> 
> > I had a query to search for certain fields which was scored using a  
> > custom script. Then I needed to take account of missing data and found  
> > that I'd need to use a filter for this. The solution query  
> > (simplified) is:
> 
> > My problem is that the 'match\_all' query will return all the data,  
> > whereas I only want data matched by my filter to be returned. By  
> > doctoring the scoring script I can ensure that filtered data is top of  
> > the list, but I'm worried about performance.
> > 
> > My original query returned only data matching my terms, but I was  
> > unable to use missing data with that. The original query is:
> 
> I'm unclear from your examples exactly what your query should do. Could  
> you explain in english what you want to achieve?
> 
> ta
> 
> clint
> 
> > {  
> > "query": {  
> > "custom\_score": {  
> > "query": {  
> > "bool": {  
> > "should": [  
> > {  
> > "term": {  
> > "id": "hi"  
> > }  
> > },  
> > {  
> > "term": {  
> > "id": "hello"  
> > }  
> > }  
> > ]  
> > }  
> > },  
> > "script": "((doc['id'].value == 'hi') ? 2 : 0)"  
> > }  
> > }  
> > }
> > 
> > Please note that this is a simplified query, I need the script rather  
> > than using a boost.
> > 
> > Is there a better solution?
> > 
> > --

--

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [October 11, 2012, 1:10pm UTC](https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303/4 "2012-10-11T13:10:11Z")

</div>

Hi Amy

> To clarify, in this example I want to return all documents where the  
> term "id" is either "hi" or "hello" or is missing altogether.  
> In the query I provided (without the filter) documents without the  
> term "id" were not returned, which was not what I wanted.  
> In the query that had the filter all documents were returned, which  
> was also not what I wanted, but I was able to bring the documents I  
> wanted to the top of the search by adding 20 to the score in the  
> filter. This is not ideal, and I'm worried about performance, even if  
> I were to change the size returned to 1.  
> Does that explain my predicament?

Yes - much clearer 🙂

The exact structure of the query depends on what else you want to do,  
but based on the description above, you could do this:

curl -XGET '[http://127.0.0.1:9200/test/test/\_search?pretty=1](http://127.0.0.1:9200/test/test/_search?pretty=1)' -d '  
{  
"query" : {  
"custom\_score" : {  
"script" : "some script goes here",  
"query" : {  
"constant\_score" : {  
"filter" : {  
"or" : [  
{ "terms" : { "id" : ["hi", "hello"] }},  
{ "missing" : { "field" : "id" }}  
]  
}  
}  
}  
}  
}  
}  
'

clint

--

---

<div class="post-metadata">

**Author:** ![Amy](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Amy](https://discuss.elastic.co/u/Amy)\
**Post date:** [October 11, 2012, 4:52pm UTC](https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303/5 "2012-10-11T16:52:43Z")

</div>

Thank you Clint, that was exactly what I was looking for!

On Thursday, October 11, 2012 2:10:22 PM UTC+1, Clinton Gormley wrote:

> Hi Amy
> 
> > To clarify, in this example I want to return all documents where the  
> > term "id" is either "hi" or "hello" or is missing altogether.  
> > In the query I provided (without the filter) documents without the  
> > term "id" were not returned, which was not what I wanted.  
> > In the query that had the filter all documents were returned, which  
> > was also not what I wanted, but I was able to bring the documents I  
> > wanted to the top of the search by adding 20 to the score in the  
> > filter. This is not ideal, and I'm worried about performance, even if  
> > I were to change the size returned to 1.  
> > Does that explain my predicament?
> 
> Yes - much clearer 🙂
> 
> The exact structure of the query depends on what else you want to do,  
> but based on the description above, you could do this:
> 
> curl -XGET '[http://127.0.0.1:9200/test/test/\_search?pretty=1](http://127.0.0.1:9200/test/test/_search?pretty=1)' -d '  
> {  
> "query" : {  
> "custom\_score" : {  
> "script" : "some script goes here",  
> "query" : {  
> "constant\_score" : {  
> "filter" : {  
> "or" : [  
> { "terms" : { "id" : ["hi", "hello"] }},  
> { "missing" : { "field" : "id" }}  
> ]  
> }  
> }  
> }  
> }  
> }  
> }  
> '
> 
> clint

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:09am UTC](https://discuss.elastic.co/t/query-for-missing-fields-using-a-score/9303/6 "2017-07-06T03:09:18Z")

</div>