# Total tokens count in all documents **URL:** \ **Category:** Elasticsearch\ **Created:** [February 25, 2016, 8:31pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776 "2016-02-25T20:31:42Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![tyka](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@tyka](https://discuss.elastic.co/u/tyka)\ **Post date:** [February 25, 2016, 8:31pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/1 "2016-02-25T20:31:42Z")
I am trying to get the total number of tokens in documents that match a query. I haven't defined any custom mapping and the field for which I want to get the token count is of type 'string'. I tried the following query, but it gives a very large number in the order of 10^20, which is not the correct answer for my dataset. > ``` > curl -XPOST 'localhost:9200/nodename/comment/_search?pretty' -d ' > { > "query": { "match_all":{} },"aggs" : {"tk_count":{ "sum" : {"script" : "_index[\"body\"].sumttf()"}}}, "size": 0 } > }' > > ``` Any idea how to get the correct count of all tokens? ( I do not need counts for each term, but the total count). ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 26, 2016, 3:55pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/2 "2016-02-26T15:55:08Z")
> [@tyka](#): > > { > "query": { "match\_all":{} },"aggs" : {"tk\_count":{ "sum" : {"script" : "\_index["body"].sumttf()"}}}, "size": 0 } > } Reading this [1] it looks like you are repeatedly (ie for every doc in the index) adding an index-level stat ( sumttf). What I suspect you might want instead is to sum the number of tokens in each doc that matches the query which could be accessed via this script: ``` doc["body"].values.size() ``` [1] [Text scoring in scripts | Elasticsearch Guide [2.2] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/2.2/modules-advanced-scripting.html#_field_statistics_3) ---
**Author:** ![tyka](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@tyka](https://discuss.elastic.co/u/tyka)\ **Post date:** [February 26, 2016, 4:13pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/3 "2016-02-26T16:13:01Z")
Thanks, that seems to be the case with sumttf(). I tried > [@Mark\_Harwood](#): > > doc["body"].values.size() and it seems to work. But for a large string field it throws CircuitBreaker exception due to large data size. Is there an efficient way to to this aggregation? Thanks! ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 26, 2016, 4:48pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/4 "2016-02-26T16:48:35Z")
Good old circuit breaker. I meant to add a warning that this would be inefficient on large indices/fields. 🙂 The most efficient alternative is to have an indexed field that stores the number of tokens in a document which can then be accessed and summed by script-free aggregations. This shifts the computation costs from query time to index time. With some special Analyzer configuration this could potentially be achieved using a custom `TokenFilter` that emits a single token that represents how many tokens (produced by earlier Tokenizers in the chain) were part of the document. "Norms" [1] are a rougher Lucene measure of field length but I'm not sure these low-level per-doc values are accessible as part of the aggregation logic. Cheers Mark [1] [https://www.elastic.co/guide/en/elasticsearch/reference/2.2/norms.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.2/norms.html) ---
**Author:** ![tyka](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@tyka](https://discuss.elastic.co/u/tyka)\ **Post date:** [February 26, 2016, 5:21pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/5 "2016-02-26T17:21:34Z")
Thanks Mark! Looks like I need to reindex my data so that I can do this queries efficiently. Thanks, Tyka ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [February 26, 2016, 6:06pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/6 "2016-02-26T18:06:19Z")
> [@Mark\_Harwood](#): > > This shifts the computation costs from query time to index time. With some special Analyzer configuration this could potentially be achieved using a custom TokenFilter that emits a single token that represents how many tokens (produced by earlier Tokenizers in the chain) were part of the document. > **[Token count datatype | Elasticsearch Guide \[2.2\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/2.2/token-count.html)** ---
**Author:** ![tyka](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@tyka](https://discuss.elastic.co/u/tyka)\ **Post date:** [February 26, 2016, 8:46pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/7 "2016-02-26T20:46:25Z")
Thanks Nik! The "token\_count" datatype will be useful for my purpose. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:13pm UTC](https://discuss.elastic.co/t/total-tokens-count-in-all-documents/42776/8 "2017-07-05T23:13:14Z")