# Elastic Query Returning Results With doc\_count Lower Than Expected

**URL:** <https://discuss.elastic.co/t/elastic-query-returning-results-with-doc-count-lower-than-expected/100254>\
**Category:** Elasticsearch\
**Created:** [September 12, 2017, 8:03pm UTC](https://discuss.elastic.co/t/elastic-query-returning-results-with-doc-count-lower-than-expected/100254 "2017-09-12T20:03:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cb2015](https://avatars.discourse-cdn.com/v4/letter/c/7993a0/32.png) [@cb2015](https://discuss.elastic.co/u/cb2015)\
**Post date:** [September 12, 2017, 8:03pm UTC](https://discuss.elastic.co/t/elastic-query-returning-results-with-doc-count-lower-than-expected/100254/1 "2017-09-12T20:03:01Z")

</div>

Hi, So I am running a query to get the number of api calls in my logs.  
When I use this query, the numbers come in too low

```
{
"query": {
"bool": {
  "must": [
    {
      "query_string": {
        "analyze_wildcard": true,
        "query": "*"
      }
    },
    {
      "range": {
        "@timestamp": {
          "gte": 1505102400000,
          "lte": 1505188799999,
          "format": "epoch_millis"
        }
      }
    }
  ],
  "must_not": []
}
  },
 "size": 0,
 "_source": {
"excludes": []
},
"aggs": {
     "2": {
  "terms": {
    "field": "api_call.keyword",
    "size": 150,
    "order": {
      "_count": "desc"
    }
  },
  "aggs": {
    "3": {
      "terms": {
        "field": "response.keyword",
        "size": 10,
        "order": {
          "_count": "desc"
        }
      }
    }
  }
}
}
}

```

As an example, when I run this, my top endpoint comes back as having 582,734 hits in a day.  
When I search my logs for that endpoint, I get 597,076 hits.

My logs having more hits than elastic for a given endpoint appears to be consistent - even the % difference between elastic and the logs stays the same.

At this point you are probably thinking that there is a problem with how I am getting data into elastic, and that the missing documents are simply not in elastic. However, when I go into kibana, and filter for an endpoint, the number I get is the exact same as what I see in my logs. In addition, when I run the query, the total number of hits matches exactly the number of hits in my logs.

The query's response also has

```
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,

```

appear before each result, so it does not seem like the issue discussed in [https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-approximate-counts](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-approximate-counts) is the cause.

Does anyone know why I am seeing this discrepancy?

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [September 13, 2017, 9:33am UTC](https://discuss.elastic.co/t/elastic-query-returning-results-with-doc-count-lower-than-expected/100254/3 "2017-09-13T09:33:24Z")

</div>

My guess is it's some tokenization discrepancy with the value used in the search request and the aggregatable value (normalizaition perhaps?).

Can you supply a reproducible example of any kind?

---

<div class="post-metadata">

**Author:** ![cb2015](https://avatars.discourse-cdn.com/v4/letter/c/7993a0/32.png) [@cb2015](https://discuss.elastic.co/u/cb2015)\
**Post date:** [September 14, 2017, 6:49pm UTC](https://discuss.elastic.co/t/elastic-query-returning-results-with-doc-count-lower-than-expected/100254/4 "2017-09-14T18:49:42Z")

</div>

Hi Mark,

So I think I figured it out.

My query returns

```
{
  "took": 412,
  "timed_out": false,
  "num_reduce_phases": 4,
  "_shards": {
    "total": 1895,
    "successful": 1895,
    "failed": 0
  },
  "hits": {
    "total": 2120635,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "2": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "3": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
             {
                "key": "200",
                "doc_count": 627050
              },
              {
                "key": "401",
                "doc_count": 31889
              },
              {
                "key": "400",
                "doc_count": 3539
              }
            ]
          },
          "key": "guest/accountInformation",
          "doc_count": 662478
        },
        {
          "3": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "200",
                "doc_count": 222928
              },
              {
                "key": "401",
                "doc_count": 23415
              },
              {
                "key": "400",
                "doc_count": 148
              }
            ]
          },
          "key": "message/myMessages",
          "doc_count": 246491
        }, ...

```

(there is more, but I don't think people would appreciate reading a 1000+ line query result)

I noticed that when I filter in "Kibana Discover" on "guest/accountInformation" I get two results -

guest/accountInformation and /guest/accountInformation, with 662,478 and 23,586 hits respectively.  
Looking at other endpoints I see that a few instances of each one has this leading '/'.

These two numbers together add up to the total that I see in my logs. So I guess my question now changes to "Is there an easy way to remove leading /'s in existing documents?"

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2017, 6:49pm UTC](https://discuss.elastic.co/t/elastic-query-returning-results-with-doc-count-lower-than-expected/100254/5 "2017-10-12T18:49:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.