# Does Text Analysis increaase CPU utilization

**URL:** <https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078>\
**Category:** Elasticsearch\
**Created:** [September 18, 2020, 9:35am UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078 "2020-09-18T09:35:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dimm](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dimm](https://discuss.elastic.co/u/dimm)\
**Post date:** [September 18, 2020, 9:35am UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078/1 "2020-09-18T09:35:55Z")

</div>

Hi,  
I want to add some analyzer to tokenize text fields in some specific way, will it have impact to ES cluster resourses utilization (CPU, ... )? If yes, how significant?

thanks

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [September 22, 2020, 7:58am UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078/2 "2020-09-22T07:58:20Z")

</div>

Hey,

that depends on your text analysis chain. If you use something like ngrams and split the text into many tokens, this will naturally take more time (and also more space on disk as it writes more data) compared to using a keyword with a single term.

I rarely care about this, as the more important question usually is, if the mapping configuration allows you to write queries that answer the question you got in a relevant fashion.

Is there any specific question hidden behind that one, that we could help with? 🙂

--Alex

---

<div class="post-metadata">

**Author:** ![dimm](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dimm](https://discuss.elastic.co/u/dimm)\
**Post date:** [September 22, 2020, 8:19am UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078/3 "2020-09-22T08:19:42Z")

</div>

Thank you for your response, Alex.  
I'm going to replace standard tokenizer with my custom:

```auto
    "tokenizer": {
      "custom_tokenizer" : {
        "type" : "char_group",
        "tokenize_on_chars" : ["whitespace", "\n", "punctuation", "symbol"]
      }
    }

```

and set this tokenizer for some specific fields.

Is it possible to predict in some way how this increase disk usage?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [September 22, 2020, 8:52am UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078/4 "2020-09-22T08:52:21Z")

</div>

I don't think you will notice a huge difference to the standard analyzer (gut feeling, nothing I tested).

That said, your best bet is to take a sample set of your data (like 10k or 100k or 500k), index them into an index with the standard tokenizer, then index into an index with the custom one, potentially run a force merge and compare the size (note, you can still end up with different segment numbers, so this is still not fully comparable, but may give you a small hint. you could also run a force merge and merge down to one segment then that would be comparable).

---

<div class="post-metadata">

**Author:** ![dimm](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dimm](https://discuss.elastic.co/u/dimm)\
**Post date:** [September 22, 2020, 8:56am UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078/5 "2020-09-22T08:56:45Z")

</div>

Thank you, thats actually I'm going to do)

---

<div class="post-metadata">

**Author:** ![dimm](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dimm](https://discuss.elastic.co/u/dimm)\
**Post date:** [September 28, 2020, 12:43pm UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078/6 "2020-09-28T12:43:54Z")

</div>

Just to summarize - in my case the difference was 1% (what is within the statistical error)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2020, 12:44pm UTC](https://discuss.elastic.co/t/does-text-analysis-increaase-cpu-utilization/249078/7 "2020-10-26T12:44:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.