# Pattern analyzer does not respect max\_token\_length
**URL:** https://discuss.elastic.co/t/pattern-analyzer-does-not-respect-max-token-length/61975
**Category:** Elasticsearch
**Created:** [October 1, 2016, 1:13am UTC](https://discuss.elastic.co/t/pattern-analyzer-does-not-respect-max-token-length/61975 "2016-10-01T01:13:54Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@rellimcire](https://discuss.elastic.co/u/rellimcire)
#### Post date: [October 1, 2016, 1:13am UTC](https://discuss.elastic.co/t/pattern-analyzer-does-not-respect-max-token-length/61975/1 "2016-10-01T01:13:54Z")
I'm testing with ES 2.3.5. My tests show that the pattern analyzer does not respect the max\_token\_length value.
First I try the standard analyzer.
```
POST /_analyze
{
"analyzer": "standard",
"text": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}
```
In the result I see the default 255 char max\_token\_length value applied.
```
{
"tokens": [
{
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"start_offset": 0,
"end_offset": 255,
"type": "",
"position": 0
},
{
"token": "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy",
"start_offset": 255,
"end_offset": 295,
"type": "",
"position": 1
}
]
}
```
I try the same thing with the pattern analyzer.
```
POST /_analyze
{
"analyzer": "pattern",
"text": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}
```
The result is one token longer than the 255 character limit.
```
{
"tokens": [
{
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy",
"start_offset": 0,
"end_offset": 295,
"type": "word",
"position": 0
}
]
}
```
I see similar failure when I explicitly set max\_token\_length or when I try to use my own pattern analyzers and tokenizers with a max\_token\_length value.
The big problem I'm trying to solve is some of my not\_analyzed data has values of length greater than 32,766 characters. This triggers a Lucene error because I get terms greater than the Lucene max length (as well discussed on [Stackoverflow UTF-8 Encoding Longer Than Max](http://stackoverflow.com/questions/24019868/utf8-encoding-is-longer-than-the-max-length-32766)). I still want some search and sort capability on these long field values, even if it is not perfect.
---
### Author:  [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [October 1, 2016, 11:08am UTC](https://discuss.elastic.co/t/pattern-analyzer-does-not-respect-max-token-length/61975/2 "2016-10-01T11:08:18Z")
Looks like a bug to me. I filed
>
The setting is documented as defaulting to 255 but we're not splitting there. I took a look at the code and didn't see anything about maximum token length either. I certainly could have missed it because it was only a quick look though.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:15pm UTC](https://discuss.elastic.co/t/pattern-analyzer-does-not-respect-max-token-length/61975/3 "2017-07-05T22:15:42Z")