# 'TokenStream contract violation' error after update to 0.90.10 **URL:** \ **Category:** Elasticsearch\ **Created:** [January 13, 2014, 3:42pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224 "2014-01-13T15:42:52Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jorjives](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jorjives/32/1706_2.png) [@jorjives](https://discuss.elastic.co/u/jorjives)\ **Post date:** [January 13, 2014, 3:42pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/1 "2014-01-13T15:42:52Z")
Hello folks, I've just updgraded to 0.90.10 from 0.90.3. When indexing I'm now getting the following message: [2014-01-13 15:34:37,116][WARN][cluster.action.shard] [Kogar] [ corpusv5][4] sending failed shard for [corpusv5][4], node[ dRuzryJqQ7eDsSj68YjFHg], [R], s[INITIALIZING], indexUUID [_na_], reason [ Failed to start shard, message [RecoveryFailedException[[corpusv5][4]: Recovery failed from [Force][QXLJK6rkQb-u7DtzaN0V\_A][inet[/10.33.159.105: 9300]] into [Kogar][dRuzryJqQ7eDsSj68YjFHg][inet[/10.33.159.105:9301]]];nested : RemoteTransportException[[Force][inet[/10.33.159.105:9300]][index/shard/ recovery/startRecovery]]; nested: RecoveryEngineException[[corpusv5][4] Phase[2] Execution failed]; nested: RemoteTransportException[[Kogar][inet[/ 10.33.159.105:9301]][index/shard/recovery/translogOps]]; nested: IndexFailedEngineException[[corpusv5][4] Index failed for [bikmo#60246]]; nested: IllegalStateException[TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.]; ]] This doesn't occur on every document being indexed and I'm not sure what is causing it. Does anybody have any ideas? Many thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f13391ff-3961-4202-8c0f-74a579affadd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f13391ff-3961-4202-8c0f-74a579affadd%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 13, 2014, 3:49pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/2 "2014-01-13T15:49:09Z")
This error is caused by an analyzer in your mapping which does no longer conform to the Lucene TokenStream API contract, which has become stricter in latest release 4.6. Probably you use a plugin with a custom analyzer? You must fix the analyzer code (i.e. call super.reset()) as it says in the error message. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY9ZNz1-tEit8KTJsQMZje%2B0rcKXGSez7F0wNDd0tJ6g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY9ZNz1-tEit8KTJsQMZje%2B0rcKXGSez7F0wNDd0tJ6g%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jorjives](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jorjives/32/1706_2.png) [@jorjives](https://discuss.elastic.co/u/jorjives)\ **Post date:** [January 13, 2014, 4:09pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/3 "2014-01-13T16:09:01Z")
> This error is caused by an analyzer in your mapping which does no longer > conform to the Lucene TokenStream API contract, which has become stricter > in latest release 4.6. Ah, thank you for the pointer. I'll investigate further in that direction. > Probably you use a plugin with a custom analyzer? As far as I'm aware, only standard analyzers are being used. We've got head, bigdesk, and hq plugins but that's it. The index has the following settings, of which I can't really see anything out of the ordinary: { "settings":{ "analysis":{ "filter": { "camel\_caps" : { "type" : "pattern\_capture", "preserve\_original" : 1, "patterns" : [ "(\p{Ll}+|\p{Lu}\p{Ll}+|\p{Lu}+)", "(\d+)" ] }, "simple\_stemmer": { "type": "stemmer", "name": "minimal\_english" }, "word\_mash": { "type": "shingle", "token\_separator": "" }, "word\_dash": { "type": "pattern\_replace", "pattern": "[^0-9a-z]+", "replacement": "-" }, "partial\_middle":{ "type":"nGram", "max\_gram":50, "min\_gram":2 } }, "analyzer":{ "search":{ "filter":[ "trim", "asciifolding", "camel\_caps", "lowercase", "simple\_stemmer", "word\_mash" ], "type":"custom", "tokenizer":"standard" }, "index": { "filter":[ "trim", "asciifolding", "lowercase", "word\_dash" ], "type":"custom", "tokenizer":"keyword" }, "suggest": { "filter": [ "trim", "asciifolding", "lowercase", "partial\_middle" ], "type": "custom", "tokenizer": "standard" } } } } } Hmmm... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/30fa5606-1ab6-4430-a21e-e553d9e9a47e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/30fa5606-1ab6-4430-a21e-e553d9e9a47e%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 13, 2014, 4:43pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/4 "2014-01-13T16:43:41Z")
I'm not sure, but the org.apache.lucene.analysis.pattern.PatternReplaceFilter source code of Lucene 4.6 looks weird, it may be causing the error. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHQpV2ohbHQd-N1hFeXoy2%3Do0-QncNzTv%2BSWMRrScgKzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHQpV2ohbHQd-N1hFeXoy2%3Do0-QncNzTv%2BSWMRrScgKzg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jorjives](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jorjives/32/1706_2.png) [@jorjives](https://discuss.elastic.co/u/jorjives)\ **Post date:** [January 13, 2014, 5:00pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/5 "2014-01-13T17:00:12Z")
Maybe a simplistic approach but: I un-installed and re-installed ES and voila: errors gone. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/374381ce-27c6-44b8-a9d9-c3c6daddbf69%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/374381ce-27c6-44b8-a9d9-c3c6daddbf69%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 13, 2014, 6:55pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/6 "2014-01-13T18:55:53Z")
I see, maybe you used some old Lucene jars by accident. Jörg On Mon, Jan 13, 2014 at 6:00 PM, Jorj Ives [jorj@bikmo.com](mailto:jorj@bikmo.com) wrote: > Maybe a simplistic approach but: > > I un-installed and re-installed ES and voila: errors gone. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHJTiar1%2Buz1BKe0EyvQ1Af4%2BYO9H1Nv6YBvA-zhm2SvQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHJTiar1%2Buz1BKe0EyvQ1Af4%2BYO9H1Nv6YBvA-zhm2SvQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Uwe\_Schindler](https://avatars.discourse-cdn.com/v4/letter/u/a8b319/32.png) [@Uwe\_Schindler](https://discuss.elastic.co/u/Uwe_Schindler)\ **Post date:** [January 13, 2014, 9:26pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/7 "2014-01-13T21:26:58Z")
Hi, In your case, StandardTokenizer was the bad guy. If you had an older version in your classpath (lucene-analyzers-common-4.5.x or whatever), this one may have been used instead the correct one of 4.6. In 4.6, together with the other contract improvements, a missing super.reset() was added to StandardTokenizer, missing in older versions: [[Apache-SVN] Diff of /lucene/dev/branches/branch\_4x/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java](http://goo.gl/M7MgdU) Uwe (who did the changes with Robert Muir) Am Montag, 13. Januar 2014 19:55:53 UTC+1 schrieb Jörg Prante: > I see, maybe you used some old Lucene jars by accident. > > Jörg > > On Mon, Jan 13, 2014 at 6:00 PM, Jorj Ives \<[jo...@bikmo.com](mailto:jo...@bikmo.com) \\>wrote: > > > Maybe a simplistic approach but: > > > > I un-installed and re-installed ES and voila: errors gone. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d4a9b695-e509-401a-a6e4-4ae16ed91e37%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d4a9b695-e509-401a-a6e4-4ae16ed91e37%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:57am UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-error-after-update-to-0-90-10/15224/8 "2017-07-06T01:57:06Z")