# How to get document length in terms of total terms/tokens - not in bytes?

**URL:** <https://discuss.elastic.co/t/how-to-get-document-length-in-terms-of-total-terms-tokens-not-in-bytes/191115>\
**Category:** Elasticsearch\
**Created:** [July 18, 2019, 1:14am UTC](https://discuss.elastic.co/t/how-to-get-document-length-in-terms-of-total-terms-tokens-not-in-bytes/191115 "2019-07-18T01:14:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![peidaqi](https://avatars.discourse-cdn.com/v4/letter/p/e274bd/32.png) [@peidaqi](https://discuss.elastic.co/u/peidaqi)\
**Post date:** [July 18, 2019, 1:14am UTC](https://discuss.elastic.co/t/how-to-get-document-length-in-terms-of-total-terms-tokens-not-in-bytes/191115/1 "2019-07-18T01:14:00Z")

</div>

I'm trying to find a way to get the total length of a document in terms of tokens.

Right now I can achieve this using the termvectors API and do a sum along all terms' ttf, but I wonder if there's a more straightforward way?

Also I'd like to get the average length of document, also in number of tokens. I can do that by running through all documents, but I wonder if there's an easier way?

Thanks

---

<div class="post-metadata">

**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\
**Post date:** [July 25, 2019, 8:30pm UTC](https://discuss.elastic.co/t/how-to-get-document-length-in-terms-of-total-terms-tokens-not-in-bytes/191115/2 "2019-07-25T20:30:14Z")

</div>

Some other pointers that may help you in your problem:

1. You can use `explain=true` parameter in your query to get more details how scores were calculated. As a part of these details, you can obtain an average length of document in this index field:

For example:

```auto
GET my_index/_search?explain=true
{
  "query" : {
  	"term": {"my_field": "fox"}
  }
}

```

produces response:

```auto
...
description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
...
{
	"value": 6.0,
	"description": "avgFieldLength",
	"details": []
},

```

1. Another way is to have a special field type [token\_count](https://www.elastic.co/guide/en/elasticsearch/reference/master/token-count.html) that will calculate tokens count for every document for the specified field. To calculate the total value of tokens across all documents you can then create a `sum` aggregation on this `token_count` field, for the average value – `avg` aggregation.

---

<div class="post-metadata">

**Author:** ![peidaqi](https://avatars.discourse-cdn.com/v4/letter/p/e274bd/32.png) [@peidaqi](https://discuss.elastic.co/u/peidaqi)\
**Post date:** [July 30, 2019, 12:43am UTC](https://discuss.elastic.co/t/how-to-get-document-length-in-terms-of-total-terms-tokens-not-in-bytes/191115/3 "2019-07-30T00:43:02Z")

</div>

Thanks. I indeed used #2 to get the avg document length. Just seems a bit odd this is not provided as part of the return value of termvectors API.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 27, 2019, 12:43am UTC](https://discuss.elastic.co/t/how-to-get-document-length-in-terms-of-total-terms-tokens-not-in-bytes/191115/4 "2019-08-27T00:43:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.