# Having problems with hit count from OR filter **URL:** \ **Category:** Elasticsearch\ **Created:** [October 24, 2014, 2:05am UTC](https://discuss.elastic.co/t/having-problems-with-hit-count-from-or-filter/20414 "2014-10-24T02:05:49Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Lenin](https://avatars.discourse-cdn.com/v4/letter/l/3bc359/32.png) [@Lenin](https://discuss.elastic.co/u/Lenin)\ **Post date:** [October 24, 2014, 2:05am UTC](https://discuss.elastic.co/t/having-problems-with-hit-count-from-or-filter/20414/1 "2014-10-24T02:05:49Z")
I have the following aggregation which show all the count by values for a particular field values. [http://localhost:8200/index1/collection1/\_search?search\_type=count](http://localhost:8200/index1/collection1/_search?search_type=count) { "aggs" : { "effects" : { "terms" : { "field" : "type" } } } } Output is { "took": 2, "timed\_out": false, "\_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 133490, "max\_score": 0, "hits": [] }, "aggregations": { "effects": { "buckets": [ { "key": "snp", "doc\_count": 112918 }, { "key": "indel", "doc\_count": 15725 }, { "key": "mnp", "doc\_count": 3751 }, { "key": "mixed", "doc\_count": 1096 } ] } } } When i count the individual count, the total tallies to 133490 (which is the total number of docs in the colleciton. But when i do the following query, i don't get the exact result count ( I am using all the possible values which returned above and converted to an OR query ) : { "query": { "filtered": { "filter": { "and": [ ``` { "query": { "filtered": { "filter": { "or": { "filters" : [ { "query": { "match": { "type": "SNP" } } }, { "query": { "match": { "type": "INS" } } }, { "query": { "match": { "type": "DEL" } } }, { "query": { "match": { "type": "COMPLEX" } } }, { "query": { "match": { "type": "MNP" } } }, { "query": { "match": { "type": "MIXED" } } } ] } } } } } ] } } } ``` } Output : { "took": 3, "timed\_out": false, "\_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 117765, "max\_score": 1, "hits": [ .....] } } As you can see the result hit count doesn't match the number of documents. When i convert the above query from a match to "terms" based one, i get the exact count. { "query": { "filtered": { "filter": { "and": [ { "query": { "filtered": { "filter": { "and" : [{ "query": { "terms": { "type": ["snp", "mixed", "indel", "mnp"] } } }] } } } } ] } } } } Is this an issue with the OR query ? Also, is there a suitable alternative with the match query where i could easily represent the above query like : { "query" : { "match" : { "type" : ["snp", "mixed", "indel", "mnp"] } } } Any help is appreciated. Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [October 27, 2014, 5:32pm UTC](https://discuss.elastic.co/t/having-problems-with-hit-count-from-or-filter/20414/2 "2014-10-27T17:32:12Z")
Hi Lenin, This looks like a bug indeed... Did you manage to nail down this issue? Could you run the same terms aggregation on the "or" query to see the distribution of terms? On Fri, Oct 24, 2014 at 4:05 AM, Lenin [lsubramanian@maverixbio.com](mailto:lsubramanian@maverixbio.com) wrote: > I have the following aggregation which show all the count by values for a > particular field values. > > [http://localhost:8200/index1/collection1/\_search?search\_type=count](http://localhost:8200/index1/collection1/_search?search_type=count) > { > "aggs" : { > "effects" : { > "terms" : { > "field" : "type" > } > } > } > } > > Output is > { > "took": 2, > "timed\_out": false, > "\_shards": { > "total": 5, > "successful": 5, > "failed": 0 > }, > "hits": { > "total": 133490, > "max\_score": 0, > "hits": > }, > "aggregations": { > "effects": { > "buckets": [ > { > "key": "snp", > "doc\_count": 112918 > }, > { > "key": "indel", > "doc\_count": 15725 > }, > { > "key": "mnp", > "doc\_count": 3751 > }, > { > "key": "mixed", > "doc\_count": 1096 > } > ] > } > } > } > > When i count the individual count, the total tallies to 133490 (which is > the total number of docs in the colleciton. > > But when i do the following query, i don't get the exact result count ( I > am using all the possible values which returned above and converted to an > OR query ) : > > { > "query": { > "filtered": { > "filter": { > "and": [ > > ``` > { > "query": { > "filtered": { > "filter": { > "or": { "filters" : [ > { > "query": { > "match": { > "type": "SNP" > } > } > }, > { > "query": { > "match": { > "type": "INS" > } > } > }, > { > "query": { > "match": { > "type": "DEL" > } > } > }, > { > "query": { > "match": { > "type": "COMPLEX" > } > } > }, > { > "query": { > "match": { > "type": "MNP" > } > } > }, > { > "query": { > "match": { > "type": "MIXED" > } > } > } > ] } > } > } > } > } > ] > } > } > } > > ``` > > } > > Output : > { > "took": 3, > "timed\_out": false, > "\_shards": { > "total": 5, > "successful": 5, > "failed": 0 > }, > "hits": { > "total": 117765, > "max\_score": 1, > "hits": [ > .....] > } > } > > As you can see the result hit count doesn't match the number of documents. > When i convert the above query from a match to "terms" based one, i get the > exact count. > { > "query": { > "filtered": { > "filter": { > "and": [ > { > "query": { > "filtered": { > "filter": { > "and" : [{ > "query": { > "terms": { > "type": ["snp", "mixed", "indel", "mnp"] > } > } > }] > } > } > } > } > ] > } > } > } > } > > Is this an issue with the OR query ? > > Also, is there a suitable alternative with the match query where i could > easily represent the above query like : > { > "query" : { > "match" : { "type" : ["snp", "mixed", "indel", "mnp"] } > } > } > > Any help is appreciated. > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5vvcwf0jw8WoOUXkrHBQF\_xqG%2BR19YA4NriNGSrkv1sA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5vvcwf0jw8WoOUXkrHBQF_xqG%2BR19YA4NriNGSrkv1sA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Lenin](https://avatars.discourse-cdn.com/v4/letter/l/3bc359/32.png) [@Lenin](https://discuss.elastic.co/u/Lenin)\ **Post date:** [October 27, 2014, 6:12pm UTC](https://discuss.elastic.co/t/having-problems-with-hit-count-from-or-filter/20414/3 "2014-10-27T18:12:52Z")
Hi Adrien, Thanks for getting back. I was able indeed fix the issue, it was a data problem in my end. But I ran into another issue with OR filter while i was able to figure the above one. I have posted it as github. > > > I have the following records > > \`\`\` javascript > { > "took": 0, > "timed\_out": f…alse, > "\_shards": { > "total": 5, > "successful": 5, > "failed": 0 > }, > "hits": { > "total": 9, > "max\_score": 1, > "hits": \[ > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "V2xbgfWwTzuNyilVGxjElw", > "\_score": 1, > "\_source": { > "firstName": "F1", > "lastName": "L1", > "age": 20 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "fnuqEGNzTHSM8tWsjX2o1w", > "\_score": 1, > "\_source": { > "firstName": "F2", > "lastName": "L1", > "age": 21 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "3TUO53bDQ22rYFcuT-oY3g", > "\_score": 1, > "\_source": { > "firstName": "F4", > "lastName": "L1", > "age": 23 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "h6zZ5WaRQbaSIHanGFpTVg", > "\_score": 1, > "\_source": { > "firstName": "F5", > "lastName": "L1", > "age": 24 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "pOLxQxb\_QXSMFG82GocYVQ", > "\_score": 1, > "\_source": { > "firstName": "F1", > "lastName": "L2", > "age": 31 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "4K7kTX32Qui4B3OGOgXahw", > "\_score": 1, > "\_source": { > "firstName": "F2", > "lastName": "L2", > "age": 32 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "HfbbfcBrRQueSTCSCWM1tg", > "\_score": 1, > "\_source": { > "firstName": "F3", > "lastName": "L2", > "age": 33 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "eqQzRvN9SqalZsqZRncb9A", > "\_score": 1, > "\_source": { > "firstName": "F3", > "lastName": "L1", > "age": 22 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "VRuZXRRWSO2PloBY-KUfcA", > "\_score": 1, > "\_source": { > "firstName": "F4", > "lastName": "L2", > "age": 34 > } > } > \] > } > } > \`\`\` > > When i query for all records where the lastName is "L1" and age 20 or lastName is L2 with age 31 using the the query : > > \`\`\` javascript > { > "query": { > "filtered": { > "filter" : { > "or": \[{ > "query": { > "filtered": { > "filter": { > "and": \[{ > "query": { > "match": { > "lastName": "L1" > } > } > }, { > "query": { > "match": { > "age": 20 > } > } > }\] > } > } > } > }, { > "query": { > "filtered": { > "filter": { > "and": \[{ > "query": { > "match": { > "lastName": "L2" > } > } > }, { > "query": { > "match": { > "age": 31 > } > } > }\] > } > } > } > }\] > } > } > } > } > \`\`\` > > I get the results correct with the following response : > > \`\`\` javascript > { > "took": 1, > "timed\_out": false, > "\_shards": { > "total": 5, > "successful": 5, > "failed": 0 > }, > "hits": { > "total": 2, > "max\_score": 1, > "hits": \[ > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "V2xbgfWwTzuNyilVGxjElw", > "\_score": 1, > "\_source": { > "firstName": "F1", > "lastName": "L1", > "age": 20 > } > }, > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "pOLxQxb\_QXSMFG82GocYVQ", > "\_score": 1, > "\_source": { > "firstName": "F1", > "lastName": "L2", > "age": 31 > } > } > \] > } > } > \`\`\` > > But the moment i add an "and" filter to the same query where still its applicable for both the records, i get only one record in the response : > > \`\`\` javascript > { > "query": { > "filtered": { > "filter": { > "and": \[{ > "query": { > "terms": { > "firstName": \[ > "f1" > \] > } > } > }\], > "or": \[{ > "query": { > "filtered": { > "filter": { > "and": \[{ > "query": { > "match": { > "lastName": "L1" > } > } > }, { > "query": { > "match": { > "age": 20 > } > } > }\] > } > } > } > }, { > "query": { > "filtered": { > "filter": { > "and": \[{ > "query": { > "match": { > "lastName": "L2" > } > } > }, { > "query": { > "match": { > "age": 31 > } > } > }\] > } > } > } > }\] > } > } > } > } > \`\`\` > > Response : > > \`\`\` javascript > { > "took": 0, > "timed\_out": false, > "\_shards": { > "total": 5, > "successful": 5, > "failed": 0 > }, > "hits": { > "total": 1, > "max\_score": 1, > "hits": \[ > { > "\_index": "testindex", > "\_type": "collection1", > "\_id": "pOLxQxb\_QXSMFG82GocYVQ", > "\_score": 1, > "\_source": { > "firstName": "F1", > "lastName": "L2", > "age": 31 > } > } > \] > } > } > \`\`\` > > It appears to be clearly an issue with the OR filter. I have other cases where OR also fails. But this appears to give a picture about what is going on. > > I don't know exactly if there is any issue with how i am using the filter/query. If there is an alternative approach where it would work well, i would appreciate the help. > Let me know if you need any more information as well. > Thanks. I have a testdata to simulate the same as well. Please let me know if you need anything more. Thanks. -Lenin On Monday, October 27, 2014 10:32:20 AM UTC-7, Adrien Grand wrote: > Hi Lenin, > > This looks like a bug indeed... Did you manage to nail down this issue? > Could you run the same terms aggregation on the "or" query to see the > distribution of terms? > > On Fri, Oct 24, 2014 at 4:05 AM, Lenin \<[lsubra...@maverixbio.com](mailto:lsubra...@maverixbio.com) > \\> wrote: > > > I have the following aggregation which show all the count by values for a > > particular field values. > > > > [http://localhost:8200/index1/collection1/\_search?search\_type=count](http://localhost:8200/index1/collection1/_search?search_type=count) > > { > > "aggs" : { > > "effects" : { > > "terms" : { > > "field" : "type" > > } > > } > > } > > } > > > > Output is > > { > > "took": 2, > > "timed\_out": false, > > "\_shards": { > > "total": 5, > > "successful": 5, > > "failed": 0 > > }, > > "hits": { > > "total": 133490, > > "max\_score": 0, > > "hits": > > }, > > "aggregations": { > > "effects": { > > "buckets": [ > > { > > "key": "snp", > > "doc\_count": 112918 > > }, > > { > > "key": "indel", > > "doc\_count": 15725 > > }, > > { > > "key": "mnp", > > "doc\_count": 3751 > > }, > > { > > "key": "mixed", > > "doc\_count": 1096 > > } > > ] > > } > > } > > } > > > > When i count the individual count, the total tallies to 133490 (which is > > the total number of docs in the colleciton. > > > > But when i do the following query, i don't get the exact result count ( > > I am using all the possible values which returned above and converted to an > > OR query ) : > > > > { > > "query": { > > "filtered": { > > "filter": { > > "and": [ > > > > ``` > > { > > "query": { > > "filtered": { > > "filter": { > > "or": { "filters" : [ > > { > > "query": { > > "match": { > > "type": "SNP" > > } > > } > > }, > > { > > "query": { > > "match": { > > "type": "INS" > > } > > } > > }, > > { > > "query": { > > "match": { > > "type": "DEL" > > } > > } > > }, > > { > > "query": { > > "match": { > > "type": "COMPLEX" > > } > > } > > }, > > { > > "query": { > > "match": { > > "type": "MNP" > > } > > } > > }, > > { > > "query": { > > "match": { > > "type": "MIXED" > > } > > } > > } > > ] } > > } > > } > > } > > } > > ] > > } > > } > > } > > > > ``` > > > > } > > > > Output : > > { > > "took": 3, > > "timed\_out": false, > > "\_shards": { > > "total": 5, > > "successful": 5, > > "failed": 0 > > }, > > "hits": { > > "total": 117765, > > "max\_score": 1, > > "hits": [ > > .....] > > } > > } > > > > As you can see the result hit count doesn't match the number of > > documents. When i convert the above query from a match to "terms" based > > one, i get the exact count. > > { > > "query": { > > "filtered": { > > "filter": { > > "and": [ > > { > > "query": { > > "filtered": { > > "filter": { > > "and" : [{ > > "query": { > > "terms": { > > "type": ["snp", "mixed", "indel", "mnp"] > > } > > } > > }] > > } > > } > > } > > } > > ] > > } > > } > > } > > } > > > > Is this an issue with the OR query ? > > > > Also, is there a suitable alternative with the match query where i could > > easily represent the above query like : > > { > > "query" : { > > "match" : { "type" : ["snp", "mixed", "indel", "mnp"] } > > } > > } > > > > Any help is appreciated. > > Thanks. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b466f820-d5cc-4a3b-a77a-79fe5aaa8ada%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/62ccd1be-d6d7-4d01-8de4-3d75f0c0880a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/62ccd1be-d6d7-4d01-8de4-3d75f0c0880a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:53am UTC](https://discuss.elastic.co/t/having-problems-with-hit-count-from-or-filter/20414/4 "2017-07-06T00:53:39Z")