# WhiteSpaceTokenizer buffer\_size
**URL:** https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387
**Category:** Elasticsearch
**Created:** [October 16, 2015, 1:36pm UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387 "2015-10-16T13:36:46Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [October 16, 2015, 1:36pm UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387/1 "2015-10-16T13:36:46Z")
My analysis conf as follows,
index :
analysis :
analyzer :
default\_index :
type : custom
tokenizer : whitespace
filter : [word\_delimiter, snowball, lowercase]
default\_search :
type : custom
tokenizer : whitespace
filter : [word\_delimiter, snowball, lowercase]
filter :
word\_delimiter :
type : word\_delimiter
preserve\_original : true
split\_on\_numerics : true
stem\_english\_possessive : false
My input text of length 793 chars contains "1675333000000088066"
(character position as 754-773) .
I wish to search the import id 1675333000000088066 but its not found . Then i paste my whole input string in kopf analysis plugin in that the entire string in processed as 256 characters chunk.Thus 1675333000000088066 is splitted in to 2 regions(3rd 256 chunk & 4th 256 chunk)
1st region 0 - 255
2nd region 255 - 510
3rd region 510 - 765 contains 16753330000
4th region 765-793 contains 00088066
On google it, i found this [link](http://stackoverflow.com/questions/27905265/what-is-buffer-size-in-keyword-tokenizer-in-elastic-search)
But this is for keyword tokenizer. How will i say to take the entire string instead of 256 char chunk while tokenizing through whitespace tokenizer ? Or any other suggestions ?
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [October 16, 2015, 7:22pm UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387/2 "2015-10-16T19:22:22Z")
Lucene's char-based tokenizers impose a hard-wired 256 length limit for words, see also [https://issues.apache.org/jira/browse/LUCENE-5785](https://issues.apache.org/jira/browse/LUCENE-5785)
See `MAX_WORD_LEN` in `org.apache.lucene.analysis.util.CharTokenizer`
Use pattern tokenizer [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html) with a whitespace character pattern as a workaround. It reads the whole input of the document for the tokenizer into RAM.
---
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [October 19, 2015, 6:29am UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387/3 "2015-10-19T06:29:33Z")
jprante,
Thanks for your reply.
---
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [February 4, 2016, 7:49am UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387/4 "2016-02-04T07:49:18Z")
@jprante
We replaced whitespace tokenizer by pattern tokenizer (pattern as whitespace).
Now while indexing we are hitting the max term size problem , got the following exception from logs
> IllegalArgumentException[Document contains at least one immense term in field="message" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[100, 111, 98, 106, 61, 61, 61, 61, 62, 123, 34, 108, 111, 99, 34, 58, 91, 123, 34, 100, 97, 116, 97, 34, 58, 123, 34, 117, 114, 108]...', original message: bytes can be at most 32766 in length; got 149542]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 149542];
any suggestions please ? FYI : we got nearly 50000 exceptions.
---
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [April 19, 2016, 6:55am UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387/5 "2016-04-19T06:55:41Z")
@jprante any suggestions ? also datanodes were severely affected by memory issues. jstat -gcutil shows constantly more than 95% memory , gc running continuously . Thus reverted back to whitespace tokenizer.
---
### Author:  [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [April 19, 2016, 11:34am UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387/6 "2016-04-19T11:34:16Z")
There is a setting on the string type that will through out tokens larger
than some size. That is your best bet to contain the exceptions but you are
accepting that they won't be in the index. I expect no one will find them
anyway because they are too long. I bet you could truncate them with a
token filter too but I don't know that offhand.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:58pm UTC](https://discuss.elastic.co/t/whitespacetokenizer-buffer-size/32387/7 "2017-07-05T22:58:04Z")