# Skip pattern-replace character filter on large tokens

**URL:** <https://discuss.elastic.co/t/skip-pattern-replace-character-filter-on-large-tokens/99730>\
**Category:** Elasticsearch\
**Created:** [September 7, 2017, 2:00pm UTC](https://discuss.elastic.co/t/skip-pattern-replace-character-filter-on-large-tokens/99730 "2017-09-07T14:00:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![drs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drs/32/8001_2.png) [@drs](https://discuss.elastic.co/u/drs)\
**Post date:** [September 7, 2017, 2:00pm UTC](https://discuss.elastic.co/t/skip-pattern-replace-character-filter-on-large-tokens/99730/1 "2017-09-07T14:00:52Z")

</div>

I'm indexing some documents that occasionally have really large tokens (greater than 300,000 characters). I don't care about searching against these large tokens, but my use of a pattern-replace char filter makes them take more than 30 minutes to be indexed.

I've tried adding a length token filter, but it looks like that gets applied after the char\_filter. Is there any way to prevent the char\_filter from analyzing these super long tokens?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [September 11, 2017, 7:32am UTC](https://discuss.elastic.co/t/skip-pattern-replace-character-filter-on-large-tokens/99730/2 "2017-09-11T07:32:13Z")

</div>

Hey,

the reason why the token filter does not help you here, is, because it is executed after the char filter. The char filter gets executed first, then the tokenizer, then the token filters.

I checked the lucene source very briefly, but I could not find anything regarding a size limitation in char filters (Note, I'm not a lucene expert).

--Alex

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2017, 7:32am UTC](https://discuss.elastic.co/t/skip-pattern-replace-character-filter-on-large-tokens/99730/3 "2017-10-09T07:32:19Z")

</div>

Closed on October 9, 2017, 7:32am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.