# How to limit token length?
**URL:** https://discuss.elastic.co/t/how-to-limit-token-length/79991
**Category:** Elasticsearch
**Created:** [March 25, 2017, 3:49pm UTC](https://discuss.elastic.co/t/how-to-limit-token-length/79991 "2017-03-25T15:49:28Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [March 25, 2017, 3:49pm UTC](https://discuss.elastic.co/t/how-to-limit-token-length/79991/1 "2017-03-25T15:49:28Z")
Analyzed or not, we want to limit length of one token in ES. Is there a way to enforce this?
E.g.
1. "Quick brown fox **SomeTextWithoutSpacesExceedingLimit**" to "Quick brown fox **SomeTextTruncatedWithConfiguredLimit**".
2. " **SomeTextWithoutSpacesExceedingLimit**" to " **SomeTextTruncatedWithConfiguredLimit**"
Here we want to truncate "SomeTextWithoutSpacesExceedingLimit" to some configurable limit.
Thank you.
---
### Author:  [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)
#### Post date: [March 25, 2017, 7:30pm UTC](https://discuss.elastic.co/t/how-to-limit-token-length/79991/2 "2017-03-25T19:30:19Z")
Hi, animageofmine
I don't understood what task are you solving.
But you can use Pattern tokenizer, or ngram tokenizer
> **[Pattern Tokenizer | Elasticsearch Guide \[5.2\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/analysis-pattern-tokenizer.html#analysis-pattern-tokenizer)**
here example, you can use java pattern to define mask of your token. This pattern divided your text to token beginning with upper litera:"pattern": "(?=\p{Upper})"
> ```
> curl -XPUT localhost:9curl -XPUT localhost:9200/test_token_upper -d '{
> "settings": {
> "analysis": {
> "analyzer": {
> "my_analyzer": {
> "tokenizer": "my_tokenizer"
> }
> },
> "tokenizer": {
> "my_tokenizer": {
> "type": "pattern",
> "pattern": "(?=\\p{Upper})"
> }
> }
> }
> }
> }'
>
> ```
and with \_analyze you can test, how behave this pattern:
> > curl -XPOST localhost:9200/test\_token\_upper/\_analyze?pretty -d '{
> > "analyzer": "my\_analyzer",
> > "text": "AaBbZzzzzz"
> > }'
---
### Author:  [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)
#### Post date: [March 25, 2017, 7:31pm UTC](https://discuss.elastic.co/t/how-to-limit-token-length/79991/3 "2017-03-25T19:31:17Z")
and as result you will take:
> ~$ curl -XPOST localhost:9200/test\_token\_upper/\_analyze?pretty -d '{
> "analyzer": "my\_analyzer",
> "text": "AaBbZzzzzz"
> }'
> {
> "tokens" : [
> {
> "token" : "Aa",
> "start\_offset" : 0,
> "end\_offset" : 2,
> "type" : "word",
> "position" : 0
> },
> {
> "token" : "Bb",
> "start\_offset" : 2,
> "end\_offset" : 4,
> "type" : "word",
> "position" : 1
> },
> {
> "token" : "Zzzzzz",
> "start\_offset" : 4,
> "end\_offset" : 10,
> "type" : "word",
> "position" : 2
> }
> ]
> }
---
### Author:  [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)
#### Post date: [March 25, 2017, 7:46pm UTC](https://discuss.elastic.co/t/how-to-limit-token-length/79991/4 "2017-03-25T19:46:31Z")
or you can use standard tokenizer which has parameter max\_token\_length, which divid your text on token with that length:
> },
> "tokenizer": {
> "my\_tokenizer": {
> "type": "standard",
> "max\_token\_length":5
> }
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [March 27, 2017, 6:16pm UTC](https://discuss.elastic.co/t/how-to-limit-token-length/79991/5 "2017-03-27T18:16:56Z")
@nugusbayevkk
Thank you. Somehow, my message was updated because I used "\<" & "\>" in my examples. Just fixed it
max\_token\_length seems to be closest to what we want, however, it seems to split the token after it reaches maximum token length. We want to truncate it since we don't care about it.
Use Case:
Sometimes, customers send in some random gibberish text that dose not make any sense:
e.g. "asdfasdf....." of may be 1 MB or some text in a unsupported language. We simply want to truncate such values instead of analyzing them.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 24, 2017, 6:17pm UTC](https://discuss.elastic.co/t/how-to-limit-token-length/79991/6 "2017-04-24T18:17:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.