# Truncate token filter splits 32bits character
**URL:** https://discuss.elastic.co/t/truncate-token-filter-splits-32bits-character/188301
**Category:** Elasticsearch
**Created:** [July 1, 2019, 11:38am UTC](https://discuss.elastic.co/t/truncate-token-filter-splits-32bits-character/188301 "2019-07-01T11:38:05Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@PaVal](https://discuss.elastic.co/u/PaVal)
#### Post date: [July 1, 2019, 11:38am UTC](https://discuss.elastic.co/t/truncate-token-filter-splits-32bits-character/188301/1 "2019-07-01T11:38:05Z")
Truncate token filter splits character in string "𝕕𝕣𝕖𝕒𝕞𝕔𝕒𝕥𝕔𝕙𝕖𝕣".
Is possible use 32bits characters in ES? Thank you.
DELETE test
PUT test
{
"settings" : {
"analysis" : {
"analyzer" : {
"analyzer\_1" : {
"tokenizer" : "standard",
"filter" : ["my\_truncate"]
}
},
```
"filter" : {
"my_truncate" : {
"type" : "truncate",
"length" : 5
}
}
}
}
```
}
GET test/\_analyze
{
"analyzer": "analyzer\_1",
"text": "\ud835\udd55\ud835\udd63\ud835\udd56\ud835\udd52\ud835\udd5e\ud835\udd54\ud835\udd52\ud835\udd65\ud835\udd54\ud835\udd59\ud835\udd56\ud835\udd63"
}
response and return 2.5 characters:
{
"tokens":[{"token":"\uD835\uDD55\uD835\uDD63\uD835","start\_offset":0,"end\_offset":24,"type":"","position":0}]
}
GET /
{
"name" : "develop",
"cluster\_name" : "elasticsearch",
"cluster\_uuid" : "MY0ymU7BTnWBRRqUprZkHw",
"version" : {
"number" : "7.2.0",
"build\_flavor" : "default",
"build\_type" : "rpm",
"build\_hash" : "508c38a",
"build\_date" : "2019-06-20T15:54:18.811730Z",
"build\_snapshot" : false,
"lucene\_version" : "8.0.0",
"minimum\_wire\_compatibility\_version" : "6.8.0",
"minimum\_index\_compatibility\_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 1, 2019, 11:51am UTC](https://discuss.elastic.co/t/truncate-token-filter-splits-32bits-character/188301/2 "2019-07-01T11:51:05Z")
I wonder if this would help you: [https://jolicode.com/blog/search-for-emoji-with-elasticsearch](https://jolicode.com/blog/search-for-emoji-with-elasticsearch) ?
---
### Author:  [@telendt](https://discuss.elastic.co/u/telendt)
#### Post date: [July 1, 2019, 2:47pm UTC](https://discuss.elastic.co/t/truncate-token-filter-splits-32bits-character/188301/3 "2019-07-01T14:47:44Z")
I believe that whenever ES (and Lucene) docs refer to characters they mean Java's 16-bit `char`s.
You can see that [TruncateTokenFilter](https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/TruncateTokenFilter.java#L52) simply sets length of the term to given value.
Unfortunately there are [no tests for multi byte characters](https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestTruncateTokenFilter.java#L32).
I think you would need to write your own TokenFilter that counts number of `chars` of the first `n` code points and uses that in `termAttribute.setLength(length)`.
---
### Author:  [@telendt](https://discuss.elastic.co/u/telendt)
#### Post date: [July 1, 2019, 3:02pm UTC](https://discuss.elastic.co/t/truncate-token-filter-splits-32bits-character/188301/4 "2019-07-01T15:02:45Z")
Actually I think that `pattern_capture` filter might get you want you want.
```auto
"cap5": {
"type": "pattern_capture",
"preserve_original": false,
"patterns": [
"^(.{1,5})"
]
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 29, 2019, 3:02pm UTC](https://discuss.elastic.co/t/truncate-token-filter-splits-32bits-character/188301/5 "2019-07-29T15:02:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.