# Stop filter problem: enablePositionIncrements=false is not supported anymore as of Lucene 4.4 as it can create broken token streams **URL:** \ **Category:** Elasticsearch\ **Created:** [September 4, 2013, 12:07pm UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457 "2013-09-04T12:07:12Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Jondow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jondow/32/2111_2.png) [@Jondow](https://discuss.elastic.co/u/Jondow)\ **Post date:** [September 4, 2013, 12:07pm UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/1 "2013-09-04T12:07:12Z")
I setup a stop filter as follows: ``` "filter_stop": { "type": "stop", "enable_position_increments":"false" } ``` However when I try run my application that then indexes my data, I get the following error: Caused by IllegalArgumentException: enablePositionIncrements=false is notsupported anymore as of Lucene 4.4 as it can create broken token streams -\>\> 40 | checkPositionIncrement in org.apache.lucene.analysis. util.FilteringTokenFilter Indeed, checking the 4.4 API docs, setEnablePositionIncrements() is deprecated so my question is: How do I get rid of underscores '\_' in a shingle filter without this > setting? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 4, 2013, 12:49pm UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/2 "2013-09-04T12:49:44Z")
Drop enable\_position\_increments parameter or set it to true. In shingle filters, you should set min\_shingle\_size to 2. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jondow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jondow/32/2111_2.png) [@Jondow](https://discuss.elastic.co/u/Jondow)\ **Post date:** [September 4, 2013, 2:16pm UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/3 "2013-09-04T14:16:47Z")
Hi Jörg, The problem is that the default is set to true, and with it set to true, my shingle filter results include underscores because of the stop filter in use, which I don't want. Traditionally the way to get rid of this was to set enablePositionIncrements to false in the stop filter. This is no longer possible, hence my predicament. ☹ On Wednesday, 4 September 2013 14:49:44 UTC+2, Jörg Prante wrote: > Drop enable\_position\_increments parameter or set it to true. > > In shingle filters, you should set min\_shingle\_size to 2. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![micherr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/micherr/32/1908_2.png) [@micherr](https://discuss.elastic.co/u/micherr)\ **Post date:** [December 19, 2013, 4:54am UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/4 "2013-12-19T04:54:57Z")
Hi Jondow, Is there any progress on the issue? среда, 4 сентября 2013 г., 17:16:47 UTC+3 пользователь Jondow написал: > Hi Jörg, > > The problem is that the default is set to true, and with it set to true, > my shingle filter results include underscores because of the stop filter in > use, which I don't want. Traditionally the way to get rid of this was to > set enablePositionIncrements to false in the stop filter. This is no longer > possible, hence my predicament. ☹ > > On Wednesday, 4 September 2013 14:49:44 UTC+2, Jörg Prante wrote: > > > Drop enable\_position\_increments parameter or set it to true. > > > > In shingle filters, you should set min\_shingle\_size to 2. > > > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/30411acd-1a9d-4332-a3bf-13e7249d91a8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/30411acd-1a9d-4332-a3bf-13e7249d91a8%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![rmuir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmuir/32/44949_2.png) [@rmuir](https://discuss.elastic.co/u/rmuir)\ **Post date:** [December 19, 2013, 5:04am UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/5 "2013-12-19T05:04:07Z")
the old disabling of position increments was bogus. for example a stop filter could remove a token and "move" a synonym from one word to another. so this option conflated two unrelated things: whether or not a "gap" should be introduced when a word is removed, and whether any existing positions (e.g. from synonyms) should be respected. in my opinion (but i have not thought it over in a while, look at the issue age) its possible to prevent the introduction of gaps while still respecting existing ones: [https://issues.apache.org/jira/browse/LUCENE-4065](https://issues.apache.org/jira/browse/LUCENE-4065) On Wed, Dec 18, 2013 at 11:54 PM, Michael Cheremuhin [micherr@gmail.com](mailto:micherr@gmail.com) wrote: > Hi Jondow, > > Is there any progress on the issue? > > среда, 4 сентября 2013 г., 17:16:47 UTC+3 пользователь Jondow написал: > > > Hi Jörg, > > > > The problem is that the default is set to true, and with it set to true, > > my shingle filter results include underscores because of the stop filter in > > use, which I don't want. Traditionally the way to get rid of this was to set > > enablePositionIncrements to false in the stop filter. This is no longer > > possible, hence my predicament. ☹ > > > > On Wednesday, 4 September 2013 14:49:44 UTC+2, Jörg Prante wrote: > > > > > Drop enable\_position\_increments parameter or set it to true. > > > > > > In shingle filters, you should set min\_shingle\_size to 2. > > > > > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/30411acd-1a9d-4332-a3bf-13e7249d91a8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/30411acd-1a9d-4332-a3bf-13e7249d91a8%40googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOdYfZVVDM6MjjS5E%2Bx68B8PXOkBRsjeZuRE8831frcS6CR7Fw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOdYfZVVDM6MjjS5E%2Bx68B8PXOkBRsjeZuRE8831frcS6CR7Fw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Eduard\_Dudar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eduard_dudar/32/2845_2.png) [@Eduard\_Dudar](https://discuss.elastic.co/u/Eduard_Dudar)\ **Post date:** [March 16, 2016, 2:10am UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/6 "2016-03-16T02:10:49Z")
Any updates from Elastic team on this? It seems that there's no reliable way to remove "\_" so far... ---
**Author:** ![1zaak](https://avatars.discourse-cdn.com/v4/letter/1/bc79bd/32.png) [@1zaak](https://discuss.elastic.co/u/1zaak)\ **Post date:** [February 21, 2017, 7:59am UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/7 "2017-02-21T07:59:49Z")
There is a 'fillter\_token' that is configurable in the [`shingle` token filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-shingle-tokenfilter.html). The default is "\_". I believe we can change it to "" so that it's empty if it encounters any stop words. Not sure if this is the best practice though. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:03pm UTC](https://discuss.elastic.co/t/stop-filter-problem-enablepositionincrements-false-is-not-supported-anymore-as-of-lucene-4-4-as-it-can-create-broken-token-streams/13457/8 "2017-07-05T22:03:02Z")