# TokenStream contract violation: close() call missing

**URL:** <https://discuss.elastic.co/t/tokenstream-contract-violation-close-call-missing/20590>\
**Category:** Elasticsearch\
**Created:** [November 6, 2014, 12:05am UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-close-call-missing/20590 "2014-11-06T00:05:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Richard\_Tier](https://avatars.discourse-cdn.com/v4/letter/r/a88e57/32.png) [@Richard\_Tier](https://discuss.elastic.co/u/Richard_Tier)\
**Post date:** [November 6, 2014, 12:05am UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-close-call-missing/20590/1 "2014-11-06T00:05:25Z")

</div>

An internal error happens when I do a "suggest" query. I get "TokenStream  
contract violation: close() call missing"

This only happens when I add `pre_filter` to the suggest body.

```
    {
        'text': 'my term',
        'suggest' : {
            'phrase' : {
                'field' : 'title',
                'direct_generator' : [
                    {
                        'field' : 'title.suggest',
                        'suggest_mode': 'always',
                        'pre_filter': 'autocomplete' // <-- this 

```

triggers the problem  
}  
]  
}  
}  
}

However, I need pre\_filter to achieve the desired outcome.

According the this thread  
[http://elasticsearch-users.115913.n3.nabble.com/TokenStream-contract-violation-error-after-update-to-0-90-10-td4047479.html](http://elasticsearch-users.115913.n3.nabble.com/TokenStream-contract-violation-error-after-update-to-0-90-10-td4047479.html) the  
problem may be caused by old jar files that are not aware of a stricter  
process flow lucene expects to be followed.

The thread suggests looking for old jars related to lucene - but the only  
ones I find are in /usr/share/elasticsearch/lib/, and they are all version  
4.10.2 - which is after when the thread says the fix was applied.

This problems affects me on elastic search version 1.3 and 1.4. I have  
tried reinstalling elasticsearch and the problem still happens.

Has anyone else had this problem?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [November 6, 2014, 11:28am UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-close-call-missing/20590/2 "2014-11-06T11:28:55Z")

</div>

Hmm, not good. What does your "autocomplete" analyzer look like? Can you  
post the full stack trace?

Mike McCandless

[http://blog.mikemccandless.com](http://blog.mikemccandless.com)

On Wed, Nov 5, 2014 at 7:05 PM, Richard Tier [rikatee@gmail.com](mailto:rikatee@gmail.com) wrote:

> An internal error happens when I do a "suggest" query. I get "TokenStream  
> contract violation: close() call missing"
> 
> This only happens when I add `pre_filter` to the suggest body.
> 
> ```
> {
> 'text': 'my term',
> 'suggest' : {
> 'phrase' : {
> 'field' : 'title',
> 'direct_generator' : [
> {
> 'field' : 'title.suggest',
> 'suggest_mode': 'always',
> 'pre_filter': 'autocomplete' // <-- this
> 
> ```
> 
> triggers the problem  
> }  
> ]  
> }  
> }  
> }
> 
> However, I need pre\_filter to achieve the desired outcome.
> 
> According the this thread  
> [http://elasticsearch-users.115913.n3.nabble.com/TokenStream-contract-violation-error-after-update-to-0-90-10-td4047479.html](http://elasticsearch-users.115913.n3.nabble.com/TokenStream-contract-violation-error-after-update-to-0-90-10-td4047479.html) the  
> problem may be caused by old jar files that are not aware of a stricter  
> process flow lucene expects to be followed.
> 
> The thread suggests looking for old jars related to lucene - but the only  
> ones I find are in /usr/share/elasticsearch/lib/, and they are all version  
> 4.10.2 - which is after when the thread says the fix was applied.
> 
> This problems affects me on Elasticsearch version 1.3 and 1.4. I have  
> tried reinstalling elasticsearch and the problem still happens.
> 
> Has anyone else had this problem?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smRc81rhAY5i%3D2CXgF-\_mw7kp-OW4DHSapjU59e9bND5Wpg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smRc81rhAY5i%3D2CXgF-_mw7kp-OW4DHSapjU59e9bND5Wpg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Richard\_Tier](https://avatars.discourse-cdn.com/v4/letter/r/a88e57/32.png) [@Richard\_Tier](https://discuss.elastic.co/u/Richard_Tier)\
**Post date:** [November 6, 2014, 8:05pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-close-call-missing/20590/3 "2014-11-06T20:05:43Z")

</div>

Thanks for reply.

The autocomplete analyzer:

```
{
    'analysis': {
      'analyzer': {
        'autocomplete': {
           'type': 'custom',
           'tokenizer': 'standard',
           'filter': [
              'filter_pair_shingle',
              'lowercase',
              'asciifolding',
           ]
        }
      },
      'filter': {
          'filter_pair_shingle':{
              'type': 'shingle'
          }
    }

```

stacktrace is not very informative really:

{u'\_shards': {u'successful': 0, u'failed': 5, u'total': 5, u'failures':  
[{u'index': u'test\_market', u'reason':  
u'BroadcastShardOperationFailedException[[test\_market][0] ]; nested:  
ElasticsearchException[failed to execute suggest]; nested:  
IllegalStateException[TokenStream contract violation: close() call  
missing]; ', u'shard': 0}, {u'index': u'test\_market', u'reason':  
u'BroadcastShardOperationFailedException[[test\_market][1] ]; nested:  
ElasticsearchException[failed to execute suggest]; nested:  
IllegalStateException[TokenStream contract violation: close() call  
missing]; ', u'shard': 1}, {u'index': u'test\_market', u'reason':  
u'BroadcastShardOperationFailedException[[test\_market][2] ]; nested:  
ElasticsearchException[failed to execute suggest]; nested:  
IllegalStateException[TokenStream contract violation: close() call  
missing]; ', u'shard': 2}, {u'index': u'test\_market', u'reason':  
u'BroadcastShardOperationFailedException[[test\_market][3] ]; nested:  
ElasticsearchException[failed to execute suggest]; nested:  
IllegalStateException[TokenStream contract violation: close() call  
missing]; ', u'shard': 3}, {u'index': u'test\_market', u'reason':  
u'BroadcastShardOperationFailedException[[test\_market][4] ]; nested:  
ElasticsearchException[failed to execute suggest]; nested:  
IllegalStateException[TokenStream contract violation: close() call  
missing]; ', u'shard': 4}]}}

On Thursday, November 6, 2014 11:29:22 AM UTC, Michael McCandless wrote:

> Hmm, not good. What does your "autocomplete" analyzer look like? Can you  
> post the full stack trace?
> 
> Mike McCandless
> 
> [http://blog.mikemccandless.com](http://blog.mikemccandless.com)
> 
> On Wed, Nov 5, 2014 at 7:05 PM, Richard Tier \<[rik...@gmail.com](mailto:rik...@gmail.com)  
> \<javascript:\>\> wrote:
> 
> > An internal error happens when I do a "suggest" query. I get "TokenStream  
> > contract violation: close() call missing"
> > 
> > This only happens when I add `pre_filter` to the suggest body.
> > 
> > ```
> > {
> > 'text': 'my term',
> > 'suggest' : {
> > 'phrase' : {
> > 'field' : 'title',
> > 'direct_generator' : [
> > {
> > 'field' : 'title.suggest',
> > 'suggest_mode': 'always',
> > 'pre_filter': 'autocomplete' // <-- this 
> > 
> > ```
> > 
> > triggers the problem  
> > }  
> > ]  
> > }  
> > }  
> > }
> > 
> > However, I need pre\_filter to achieve the desired outcome.
> > 
> > According the this thread  
> > [http://elasticsearch-users.115913.n3.nabble.com/TokenStream-contract-violation-error-after-update-to-0-90-10-td4047479.html](http://elasticsearch-users.115913.n3.nabble.com/TokenStream-contract-violation-error-after-update-to-0-90-10-td4047479.html) the  
> > problem may be caused by old jar files that are not aware of a stricter  
> > process flow lucene expects to be followed.
> > 
> > The thread suggests looking for old jars related to lucene - but the only  
> > ones I find are in /usr/share/elasticsearch/lib/, and they are all version  
> > 4.10.2 - which is after when the thread says the fix was applied.
> > 
> > This problems affects me on Elasticsearch version 1.3 and 1.4. I have  
> > tried reinstalling elasticsearch and the problem still happens.
> > 
> > Has anyone else had this problem?
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a22f3427-f6ac-4965-a27a-4879d3e6c851%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c8e027ec-1126-4ad2-9620-c2a3f01f2a54%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c8e027ec-1126-4ad2-9620-c2a3f01f2a54%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [November 7, 2014, 12:21pm UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-close-call-missing/20590/4 "2014-11-07T12:21:20Z")

</div>

On Thu, Nov 6, 2014 at 3:05 PM, Richard Tier [rikatee@gmail.com](mailto:rikatee@gmail.com) wrote:

> Thanks for reply.
> 
> The autocomplete analyzer:
> 
> ```
> {
> 'analysis': {
> 'analyzer': {
> 'autocomplete': {
> 'type': 'custom',
> 'tokenizer': 'standard',
> 'filter': [
> 'filter_pair_shingle',
> 'lowercase',
> 'asciifolding',
> ]
> }
> },
> 'filter': {
> 'filter_pair_shingle':{
> 'type': 'shingle'
> }
> }
> 
> ```
> 
> stacktrace is not very informative really:

Hmm indeed this is not very helpful, but I strongly suspect this is a bug  
inside Elasticsearch in how it interacts with Lucene's Analyzer/TokenStream  
API. Could you open an issue on github? Thanks.

> {u'\_shards': {u'successful': 0, u'failed': 5, u'total': 5, u'failures':  
> [{u'index': u'test\_market', u'reason':  
> u'BroadcastShardOperationFailedException[[test\_market][0] ]; nested:  
> ElasticsearchException[failed to execute suggest]; nested:  
> IllegalStateException[TokenStream contract violation: close() call  
> missing]; ', u'shard': 0}, {u'index': u'test\_market', u'reason':  
> u'BroadcastShardOperationFailedException[[test\_market][1] ]; nested:  
> ElasticsearchException[failed to execute suggest]; nested:  
> IllegalStateException[TokenStream contract violation: close() call  
> missing]; ', u'shard': 1}, {u'index': u'test\_market', u'reason':  
> u'BroadcastShardOperationFailedException[[test\_market][2] ]; nested:  
> ElasticsearchException[failed to execute suggest]; nested:  
> IllegalStateException[TokenStream contract violation: close() call  
> missing]; ', u'shard': 2}, {u'index': u'test\_market', u'reason':  
> u'BroadcastShardOperationFailedException[[test\_market][3] ]; nested:  
> ElasticsearchException[failed to execute suggest]; nested:  
> IllegalStateException[TokenStream contract violation: close() call  
> missing]; ', u'shard': 3}, {u'index': u'test\_market', u'reason':  
> u'BroadcastShardOperationFailedException[[test\_market][4] ]; nested:  
> ElasticsearchException[failed to execute suggest]; nested:  
> IllegalStateException[TokenStream contract violation: close() call  
> missing]; ', u'shard': 4}]}}

Mike McCandless

[http://blog.mikemccandless.com](http://blog.mikemccandless.com)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smRf0V0rRUU9\_KWTcK07K8kFfGtC9wdib2JWzP4qBDwS1Gw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smRf0V0rRUU9_KWTcK07K8kFfGtC9wdib2JWzP4qBDwS1Gw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:51am UTC](https://discuss.elastic.co/t/tokenstream-contract-violation-close-call-missing/20590/5 "2017-07-06T00:51:31Z")

</div>