# Sanitize a text for indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [March 12, 2015, 8:52am UTC](https://discuss.elastic.co/t/sanitize-a-text-for-indexing/22629 "2015-03-12T08:52:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Bernhard\_Berger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bernhard_berger/32/833_2.png) [@Bernhard\_Berger](https://discuss.elastic.co/u/Bernhard_Berger)\ **Post date:** [March 12, 2015, 8:52am UTC](https://discuss.elastic.co/t/sanitize-a-text-for-indexing/22629/1 "2015-03-12T08:52:10Z")
Hi, while indexing various comments from Facebook I sometimes get Exceptions: IllegalArgumentException: Document contains at least one immense term... Is it possible to sanitize a text for indexing in Elasticsearch so it doesn't throw these Exceptions? Maybe there is a Filter to remove too-long Unicode terms? For details about the failing documents, see my (unanswered) Stackoverflow question: [http://stackoverflow.com/questions/28941570/remove-long-unicode-terms-from-string-in-java](http://stackoverflow.com/questions/28941570/remove-long-unicode-terms-from-string-in-java) (I fear to break another Elasticsearch-based (Maillist) crawler, so I better don't write the failing doc text here ๐Ÿ˜‰ ) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/93a5ed0d-6486-48b4-a228-1aff47d14ce0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/93a5ed0d-6486-48b4-a228-1aff47d14ce0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [March 12, 2015, 9:03am UTC](https://discuss.elastic.co/t/sanitize-a-text-for-indexing/22629/2 "2015-03-12T09:03:49Z")
See > **[Length token filter | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-length-tokenfilter.html)** -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Thu, Mar 12, 2015 at 10:52 AM, Bernhard Berger \< [bernhardberger3456@gmail.com](mailto:bernhardberger3456@gmail.com)\> wrote: > Hi, > > while indexing various comments from Facebook I sometimes get Exceptions: > > IllegalArgumentException: Document contains at least one immense term... > > Is it possible to sanitize a text for indexing in Elasticsearch so it doesn't throw these Exceptions? Maybe there is a Filter to remove too-long Unicode terms? > > For details about the failing documents, see my (unanswered) Stackoverflow question: [encoding - Remove long unicode terms from String in Java - Stack Overflow](http://stackoverflow.com/questions/28941570/remove-long-unicode-terms-from-string-in-java) > (I fear to break another Elasticsearch-based (Maillist) crawler, so I better don't write the failing doc text here ๐Ÿ˜‰ ) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/93a5ed0d-6486-48b4-a228-1aff47d14ce0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/93a5ed0d-6486-48b4-a228-1aff47d14ce0%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/93a5ed0d-6486-48b4-a228-1aff47d14ce0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/93a5ed0d-6486-48b4-a228-1aff47d14ce0%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtqBSYcM9oFRa%3DGsWeafzHsE%3DSVMSa6H9e1aVfDbS2q%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtqBSYcM9oFRa%3DGsWeafzHsE%3DSVMSa6H9e1aVfDbS2q%3Dg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Bernhard\_Berger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bernhard_berger/32/833_2.png) [@Bernhard\_Berger](https://discuss.elastic.co/u/Bernhard_Berger)\ **Post date:** [March 12, 2015, 9:59am UTC](https://discuss.elastic.co/t/sanitize-a-text-for-indexing/22629/3 "2015-03-12T09:59:26Z")
On 12.03.15 10:03, Itamar Syn-Hershko wrote: > See > [Length token filter | Elasticsearch Guide [8.11] | Elastic](http://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-length-tokenfilter.html) Unfortunately the length token filter also doesn't filter out these immense terms. See my example from [Elasticsearch error ยท GitHub](https://gist.github.com/Hocdoc/68b5fcf8819a51816b53) : I have created a length filter for terms greater than 5000 (characters? bytes?) but still get the exception when using the icu\_normalizer : |IllegalArgumentException: Document contains at least one immense term in field="message" (whose UTF8 encoding is longer than the max length32766),| ( length of this message value is 3728 Bytes UTF8-encoded) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5501637E.2070400%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5501637E.2070400%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:27am UTC](https://discuss.elastic.co/t/sanitize-a-text-for-indexing/22629/4 "2017-07-06T00:27:16Z")