# Discrepancy in using Top Hits with Query **URL:** \ **Category:** Elasticsearch\ **Created:** [March 25, 2019, 1:09pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754 "2019-03-25T13:09:56Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![hell\_abhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hell_abhi/32/42709_2.png) [@hell\_abhi](https://discuss.elastic.co/u/hell_abhi)\ **Post date:** [March 25, 2019, 1:09pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/1 "2019-03-25T13:09:56Z")
**Note** - I moved all the results to Pastebin, due to the word count limit of this site, sorry for the inconvenience. When I am running the following query, for grouping by `creator_cust_id` in a specific date range: ``` { "from":0, "size":0, "query":{ "bool":{ "must":[ { "range":{ "lead_created_time":{ "from":"2019-02-01 00:00:00", "to":"2019-03-01 00:00:00", "format":"yyyy-MM-dd HH:mm:ss", "boost":1.0 } } } ], "adjust_pure_negative":true, "boost":1.0 } }, "aggregations":{ "creator_cust_id":{ "terms":{ "field":"creator_cust_id.keyword", "size":1000, "min_doc_count":1, "shard_min_doc_count":0, "show_term_doc_count_error":false, "order":[ { "_count":"desc" }, { "_key":"asc" } ] }, "aggregations":{ "data_fields":{ "top_hits":{ "from":0, "size":1, "version":false, "explain":false, "_source":{ "includes":[ "creator.agent_name", "creator.agent_team" ], "excludes":[ ] } } } } } } } ``` I get the proper result with all the desired data (`agent_name`, `agent_team`), which is: [https://pastebin.com/UgFeeK1a](https://pastebin.com/UgFeeK1a) But, when I add a query to get the specific `creator_cust_id`, I get proper data in one case ( **1000013335** ) and don't get the data in another case ( **1107221321** ). Here are the respective queries with the result: **Getting desired data** ``` { "from": 0, "size": 0, "query": { "bool": { "must": [ { "range": { "lead_created_time": { "from": "2019-02-01 00:00:00", "to": "2019-03-01 00:00:00", "format": "yyyy-MM-dd HH:mm:ss", "boost": 1 } } }, { "match": { "creator_cust_id": { "query": "1000013335", "operator": "OR", "prefix_length": 0, "max_expansions": 50, "fuzzy_transpositions": true, "lenient": false, "zero_terms_query": "NONE", "auto_generate_synonyms_phrase_query": true, "boost": 1 } } } ], "adjust_pure_negative": true, "boost": 1 } }, "aggregations": { "creator_cust_id": { "terms": { "field": "creator_cust_id.keyword", "size": 1000, "min_doc_count": 1, "shard_min_doc_count": 0, "show_term_doc_count_error": false, "order": [ { "_count": "desc" }, { "_key": "asc" } ] }, "aggregations": { "data_fields": { "top_hits": { "from": 0, "size": 1, "version": false, "explain": false, "_source": { "includes": [ "creator.agent_name", "creator.agent_team" ], "excludes": [] } } } } } } } ``` **Result** : [https://pastebin.com/AwJxLb91](https://pastebin.com/AwJxLb91) **Not getting desired data** ``` { "from": 0, "size": 0, "query": { "bool": { "must": [ { "range": { "lead_created_time": { "from": "2019-02-01 00:00:00", "to": "2019-03-01 00:00:00", "format": "yyyy-MM-dd HH:mm:ss", "boost": 1 } } }, { "match": { "creator_cust_id": { "query": "1107221321", "operator": "OR", "prefix_length": 0, "max_expansions": 50, "fuzzy_transpositions": true, "lenient": false, "zero_terms_query": "NONE", "auto_generate_synonyms_phrase_query": true, "boost": 1 } } } ], "adjust_pure_negative": true, "boost": 1 } }, "aggregations": { "creator_cust_id": { "terms": { "field": "creator_cust_id.keyword", "size": 1000, "min_doc_count": 1, "shard_min_doc_count": 0, "show_term_doc_count_error": false, "order": [ { "_count": "desc" }, { "_key": "asc" } ] }, "aggregations": { "data_fields": { "top_hits": { "from": 0, "size": 1, "version": false, "explain": false, "_source": { "includes": [ "creator.agent_name", "creator.agent_team" ], "excludes": [] } } } } } } } ``` **Result** : [https://pastebin.com/Pp9sMTy3](https://pastebin.com/Pp9sMTy3) Notice the empty `source` field here. I am not able to understand, why I am getting such a discrepancy, as I have used the query in a lot of aggregations, without any error till now. I tried it by using `Filter Aggregation` and it worked properly, but I can't use `Filter Aggregation` in my case. What's the logic behind this difference? How can I solve it using `query`? I know it won't make any difference, but I am using Java High-Level Rest Client, and these queries are made using that only. Do ask in the comments for any information that is missing, or need explanation. ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 25, 2019, 3:07pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/2 "2019-03-25T15:07:52Z")
I think it would be interesting to see what is the output of those queries without the aggregations. They might give light to what you are seeing. ---
**Author:** ![hell\_abhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hell_abhi/32/42709_2.png) [@hell\_abhi](https://discuss.elastic.co/u/hell_abhi)\ **Post date:** [March 25, 2019, 3:19pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/3 "2019-03-25T15:19:26Z")
Thanks for your reply, I will check them out and update with the same! ---
**Author:** ![hell\_abhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hell_abhi/32/42709_2.png) [@hell\_abhi](https://discuss.elastic.co/u/hell_abhi)\ **Post date:** [March 25, 2019, 3:42pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/4 "2019-03-25T15:42:23Z")
Here are the queries and respective result: Query for **1000013335** : [https://pastebin.com/GTLmySZx](https://pastebin.com/GTLmySZx) Result : [https://pastebin.com/bzDsQTim](https://pastebin.com/bzDsQTim) Query for **1107221321** : [https://pastebin.com/ghdA0nuJ](https://pastebin.com/ghdA0nuJ) Result : [https://pastebin.com/bzhSPMya](https://pastebin.com/bzhSPMya) ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 25, 2019, 3:46pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/5 "2019-03-25T15:46:09Z")
As you see in the result of your second query, the first element does not contain a creator, maybe it is missing in your data. The top hits aggregator is taking the value for that element, therefore it is empty. ---
**Author:** ![hell\_abhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hell_abhi/32/42709_2.png) [@hell\_abhi](https://discuss.elastic.co/u/hell_abhi)\ **Post date:** [March 25, 2019, 3:51pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/7 "2019-03-25T15:51:52Z")
Then why does the data show up, when I am using the general aggregation (without any `creator_cust_id` query)? ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 25, 2019, 3:57pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/8 "2019-03-25T15:57:59Z")
Your query is filtering by range: ```auto "range": { "lead_created_time": { "from": "2019-02-01 00:00:00", "to": "2019-03-01 00:00:00", "format": "yyyy-MM-dd HH:mm:ss", "boost": 1 } } ``` and by creator\_cust\_id: ```auto "match": { "creator_cust_id": { "query": "1107221321", "operator": "OR", "prefix_length": 0, "max_expansions": 50, "fuzzy_transpositions": true, "lenient": false, "zero_terms_query": "NONE", "auto_generate_synonyms_phrase_query": true, "boost": 1 } } ``` But it is not filtering out elements that do not have: ```auto creator.agent_name creator.agent_team ``` So the document matches the query but it contains an empty creator object. Then the top level aggregation uses `creator_cust_id.keyword` and then you have the top hits aggregator. The result is what it is expected, there is no place where you exclude documents with empty creator object. ---
**Author:** ![hell\_abhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hell_abhi/32/42709_2.png) [@hell\_abhi](https://discuss.elastic.co/u/hell_abhi)\ **Post date:** [March 25, 2019, 4:24pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/9 "2019-03-25T16:24:06Z")
Yeah I got the answer I was looking for, thanks for your help. Wanted to ask one thing, is there a way to get results where `agent_name` and `agent_team` are non empty? One way I think of is with using a mixture of aggregation of `creator_cust_id` + `agent_name`. Is there a better way to do it? ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 25, 2019, 4:59pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/10 "2019-03-25T16:59:22Z")
I guess you can you the `exists` query to filter out those documents: [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html) If you want to be part of your query but filter them from the aggregation you might be able to do it with the filter aggregation: [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 22, 2019, 4:59pm UTC](https://discuss.elastic.co/t/discrepancy-in-using-top-hits-with-query/173754/11 "2019-04-22T16:59:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.