# Understanding doc and docCount values in explain response

**URL:** <https://discuss.elastic.co/t/understanding-doc-and-doccount-values-in-explain-response/158383>\
**Category:** Elasticsearch\
**Created:** [November 27, 2018, 4:41pm UTC](https://discuss.elastic.co/t/understanding-doc-and-doccount-values-in-explain-response/158383 "2018-11-27T16:41:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ParitoshBh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paritoshbh/32/22487_2.png) [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\
**Post date:** [November 27, 2018, 4:41pm UTC](https://discuss.elastic.co/t/understanding-doc-and-doccount-values-in-explain-response/158383/1 "2018-11-27T16:41:19Z")

</div>

Here's the sample response after running query with `explain` set to `true`,

```php
[0] => Array
    (
        [value] => 24.375515
        [description] => score(doc=1115115,freq=1.0 = termFreq=1.0), product of:
        [details] => Array
            (
                [0] => Array
                    (
                        [value] => 3
                        [description] => boost
                        [details] => Array
                            (
                            )

                    )

                [1] => Array
                    (
                        [value] => 7.09822
                        [description] => idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
                        [details] => Array
                            (
                                [0] => Array
                                    (
                                        [value] => 976
                                        [description] => docFreq
                                        [details] => Array
                                            (
                                            )

                                    )

                                [1] => Array
                                    (
                                        [value] => 1181380
                                        [description] => docCount
                                        [details] => Array
                                            (
                                            )

                                    )

                            )

                    )

                [2] => Array
                    (
                        [value] => 1.1446774
                        [description] => tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:
                        [details] => Array
                            (
                                [0] => Array
                                    (
                                        [value] => 1
                                        [description] => termFreq=1.0
                                        [details] => Array
                                            (
                                            )

                                    )

                                [1] => Array
                                    (
                                        [value] => 1.2
                                        [description] => parameter k1
                                        [details] => Array
                                            (
                                            )

                                    )

                                [2] => Array
                                    (
                                        [value] => 0.75
                                        [description] => parameter b
                                        [details] => Array
                                            (
                                            )

                                    )

                                [3] => Array
                                    (
                                        [value] => 5.788349
                                        [description] => avgFieldLength
                                        [details] => Array
                                            (
                                            )

                                    )

                                [4] => Array
                                    (
                                        [value] => 4
                                        [description] => fieldLength
                                        [details] => Array
                                            (
                                            )

                                    )

                            )

                    )

            )

    )

```

Within this subset, I am trying to understand,

1. How `doc` and `docCount` are being calculated?
2. What's the difference between these 2 fields?
3. What impact does `docs.deleted` field has in its calculation considering the values changes if the index contains some deleted documents?

Any help will be appreciated!

---

<div class="post-metadata">

**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\
**Post date:** [November 30, 2018, 10:47am UTC](https://discuss.elastic.co/t/understanding-doc-and-doccount-values-in-explain-response/158383/2 "2018-11-30T10:47:47Z")

</div>

hey, lemme try to explain:

- `doc` is the internal docID. it's totally irrelevant to you it's the Nth document in a segment.
- `docCount` is the total number of documents that have at least one term in the field. For simplicity you can think of it as the number of docs in you index
- `docs.deleted` is the number of documents marked as deleted. We take them still into account when we score ie. `docCount` will contain deleted docs.

hope this helps

---

<div class="post-metadata">

**Author:** ![ParitoshBh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paritoshbh/32/22487_2.png) [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\
**Post date:** [December 3, 2018, 3:58pm UTC](https://discuss.elastic.co/t/understanding-doc-and-doccount-values-in-explain-response/158383/3 "2018-12-03T15:58:37Z")

</div>

It does help! Thanks 🙂

Couple of follow up question,

1. I am assuming then that values for `docCount` for each field will not change for a static index, irrespective of the query being executed. Is that a correct assumption?
2. Is there a way to pull out `docCount` per field or is it something that's accessible/visible through explain api only?
3. Does `docFreq` imply number of documents with successful hits (or term appears in that field)? If so for a simple match query, like following,

```auto
POST /development_en/catalog/_search?search_type=dfs_query_then_fetch&preference=local_testing&filter_path=hits.total,hits.hits._explanation
{
  "size": 1,
  "explain": true,
  "query": {
    "match": {
      "product.name": "glove"
    }
  }
}

```

shouldn't `hits.total` match `docFreq` ? I get the following response,

```auto
 {
   "hits": {
     "total": 660,
     "hits": [
       {
         "_explanation": {
           "value": 6.9444885,
           "description": "sum of:",
           "details": [
             {
               "value": 6.9444885,
               "description": "weight(product.name:glove in 690378) [PerFieldSimilarity], result of:",
               "details": [
                 {
                   "value": 6.9444885,
                   "description": "score(doc=690378,freq=1.0 = termFreq=1.0\n), product of:",
                   "details": [
                     {
                       "value": 5.3614783,
                       "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                       "details": [
                         {
                           "value": 5607,
                           "description": "docFreq",
                           "details": []
                         },
                         {
                           "value": 1194619,
                           "description": "docCount",
                           "details": []
                         }
                       ]
                     },
                     {
                       "value": 1.2952563,
                       "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                       "details": [
                         {
                           "value": 1,
                           "description": "termFreq=1.0",
                           "details": []
                         },
                         {
                           "value": 1.2,
                           "description": "parameter k1",
                           "details": []
                         },
                         {
                           "value": 0.75,
                           "description": "parameter b",
                           "details": []
                         },
                         {
                           "value": 5.7816,
                           "description": "avgFieldLength",
                           "details": []
                         },
                         {
                           "value": 2.56,
                           "description": "fieldLength",
                           "details": []
                         }
                       ]
                     }
                   ]
                 }
               ]
             },
             {
               "value": 0,
               "description": "match on required clause, product of:",
               "details": [
                 {
                   "value": 0,
                   "description": "# clause",
                   "details": []
                 },
                 {
                   "value": 1,
                   "description": "_type:catalog, product of:",
                   "details": [
                     {
                       "value": 1,
                       "description": "boost",
                       "details": []
                     },
                     {
                       "value": 1,
                       "description": "queryNorm",
                       "details": []
                     }
                   ]
                 }
               ]
             }
           ]
         }
       }
     ]
   }
 }

```

Thanks again for you time. I am trying to wrap my head around some of the numbers that Elasticsearch uses!

---

<div class="post-metadata">

**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\
**Post date:** [December 3, 2018, 7:41pm UTC](https://discuss.elastic.co/t/understanding-doc-and-doccount-values-in-explain-response/158383/4 "2018-12-03T19:41:33Z")

</div>

> [@ParitoshBh](#):
>
> Iam assuming then that values for `docCount` for each field will not change for a static index, irrespective of the query being executed. Is that a correct assumption?

Yes that is correct.

> [@ParitoshBh](#):
>
> Is there a way to pull out `docCount` per field or is it something that's accessible/visible through explain api only?

not at this point

> [@ParitoshBh](#):
>
> Does `docFreq` imply number of documents with successful hits (or term appears in that field)? If so for a simple match query, like following,

I think the confusion comes from `docFreq` being per shard and `hits.total` is hits across shards.

hope that makes sense

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 31, 2018, 7:41pm UTC](https://discuss.elastic.co/t/understanding-doc-and-doccount-values-in-explain-response/158383/5 "2018-12-31T19:41:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.