# How to compute token counts efficiently?

**URL:** <https://discuss.elastic.co/t/how-to-compute-token-counts-efficiently/26737>\
**Category:** Elasticsearch\
**Created:** [August 3, 2015, 4:57pm UTC](https://discuss.elastic.co/t/how-to-compute-token-counts-efficiently/26737 "2015-08-03T16:57:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![lucaw](https://avatars.discourse-cdn.com/v4/letter/l/bc79bd/32.png) [@lucaw](https://discuss.elastic.co/u/lucaw)\
**Post date:** [August 3, 2015, 4:57pm UTC](https://discuss.elastic.co/t/how-to-compute-token-counts-efficiently/26737/1 "2015-08-03T16:57:03Z")

</div>

In order to compute a custom script score function I'm working on I need to be able to retrieve the token counts for a collection of fields in my documents. I know that it's difficult to retrieve the field length norm value (or at least I couldn't find any way of doing so) and that instead it's recommend to use the [token\_count type](https://www.elastic.co/guide/en/elasticsearch/reference/1.6/mapping-core-types.html) as in the following example mapping:

```
"doc": {
    "properties": {
      "title": {
        "type": "string",
        "analyzer" : "my_analyzer",
        "fields": {
          "token_count": {
            "type" : "token_count",
            "store" : "yes",
            "analyzer" : "my_analyzer"
          }
        }
      }
   }
}

```

This solution works fairly well for me with the downside that since the token\_count field has to be reanalyzed it seems to dramatically slow down the speed of indexing (since I'm doing this to almost every field and there are many fields indexing appears to take twice as long). Is there anyway to make this process more efficient? In particular I'm always using the same analyzer for the token\_count sub-field as I am for it's super field, can these two not share information in some way rather than having to do the same things twice?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:57pm UTC](https://discuss.elastic.co/t/how-to-compute-token-counts-efficiently/26737/2 "2017-07-05T23:57:39Z")

</div>