# Tokens offset issue

**URL:** <https://discuss.elastic.co/t/tokens-offset-issue/307737>\
**Category:** Elasticsearch\
**Created:** [June 21, 2022, 9:20am UTC](https://discuss.elastic.co/t/tokens-offset-issue/307737 "2022-06-21T09:20:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![itaydvir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itaydvir/32/10143_2.png) [@itaydvir](https://discuss.elastic.co/u/itaydvir)\
**Post date:** [June 21, 2022, 9:20am UTC](https://discuss.elastic.co/t/tokens-offset-issue/307737/1 "2022-06-21T09:20:23Z")

</div>

Hello,  
I'm upgrading from Elastic 6.8.1 to 7.8.1 (tested on 8.2.3 as well) and getting the following error:

> startOffset must be non-negative, and endOffset must be \>= startOffset, and offsets must not go backwards

It happens when I have a text with synonym and a word with delimiter char in it.  
the change I had to do in my analyzer filters is moving the `word_delimiter` from being before the synonyms filter to be after.  
so now i have:

```auto
"analyzer": {
        "stemmed_en": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "synonym_en",
            "word_delimiter",
            "stemmer_en"
          ]
        }

```

In addition, I have a mapping for a `content` field:

```auto
"content": {
        "type": "text",
        "analyzer": "stemmed_en",
        "norms": false
      }

```

For example, if I try to index the following document, and email has a synonym [email,e mail]:

```auto
POST index_name/_doc/12345
{
    "content": "email abc@def.com"
}

```

I'm getting the following error:

> "type": "illegal\_argument\_exception",  
> "reason": "startOffset must be non-negative, and endOffset must be \>= startOffset, and offsets must not go backwards startOffset=6,endOffset=17,lastStartOffset=14 for field 'content'"

Using the analyze API I get the following tokens:

```auto
{
    "tokens": [
        {
            "token": "email",
            "start_offset": 0,
            "end_offset": 5,
            "type": "word",
            "position": 0
        },
        {
            "token": "e",
            "start_offset": 0,
            "end_offset": 5,
            "type": "SYNONYM",
            "position": 0
        },
        {
            "token": "abc",
            "start_offset": 6,
            "end_offset": 9,
            "type": "word",
            "position": 1
        },
        {
            "token": "def",
            "start_offset": 10,
            "end_offset": 13,
            "type": "word",
            "position": 2
        },
        {
            "token": "com",
            "start_offset": 14,
            "end_offset": 17,
            "type": "word",
            "position": 3
        },
        {
            "token": "mail",
            "start_offset": 6,
            "end_offset": 17,
            "type": "SYNONYM",
            "position": 3
        }
    ]
}

```

So its very clear that the synonym offset is getting mixed with the word delimiting.  
can you help me understand whats the cause and how to fix it?

- note: I have also tried using `word_delimiter_graph` instead of `word_delimiter` but got the same error.

Thanks

---

<div class="post-metadata">

**Author:** ![itaydvir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itaydvir/32/10143_2.png) [@itaydvir](https://discuss.elastic.co/u/itaydvir)\
**Post date:** [June 23, 2022, 8:50am UTC](https://discuss.elastic.co/t/tokens-offset-issue/307737/2 "2022-06-23T08:50:34Z")

</div>

looks like a bug in synonyms filter:

> <https://github.com/elastic/elasticsearch/issues/87894>
>
> \### Elasticsearch Version
> 
> 8.2.3
> 
> \### Installed Plugins
> 
> none
> 
> \### Java …Version
> 
> \_bundled\_
> 
> \### OS Version
> 
> Darwin 20.6.0 Darwin Kernel Version 20.6.0
> 
> \### Problem Description
> 
> The bug happens when indexing a text with a synonym and a word containing delimiter char in it (email for example). such text will throw the following error:
> \> startOffset must be non-negative, and endOffset must be \>= startOffset, and offsets must not go backwards startOffset=6,endOffset=17,lastStartOffset=14 for field 'content'
> 
> it happens with both \`word\_delimiter\` and \`word\_delimiter\_graph\`.
> see the given example in \`Steps to Reproduce\`.
> 
> I have tested it on versions \`7.8.1\` & \`8.2.3\`.
> 
> \### Steps to Reproduce
> 
> 1. create the index with mapping:
> \`\`\`
> PUT /text\_index
> {
> "mappings": {
> "properties": {
> "content": {
> "type": "text",
> "norms": false,
> "analyzer": "stemmed\_en"
> }
> }
> },
> "settings": {
> "index": {
> "number\_of\_shards": "1",
> "analysis": {
> "filter": {
> "synonym\_en": {
> "type": "synonym",
> "synonyms": \["email,e mail" \]
> },
> "stemmer\_en": {
> "type": "stemmer",
> "language": "english"
> }
> },
> "analyzer": {
> "stemmed\_en": {
> "filter": \[
> "lowercase",
> "synonym\_en",
> "word\_delimiter\_graph",
> "stemmer\_en"
> \],
> "type": "custom",
> "tokenizer": "whitespace"
> }
> }
> },
> "number\_of\_replicas": "1"
> }
> }
> }
> \`\`\`
> 
> 2. Index a document:
> \`\`\`
> POST /text\_index/\_doc
> {
> "content": "email abc@def.com"
> }
> \`\`\`
> 
> 3. the following error will be thrown:
> \`\`\`
> {
> "error": {
> "root\_cause": \[
> {
> "type": "illegal\_argument\_exception",
> "reason": "startOffset must be non-negative, and endOffset must be \>= startOffset, and offsets must not go backwards startOffset=6,endOffset=17,lastStartOffset=14 for field 'content'"
> }
> \],
> "type": "illegal\_argument\_exception",
> "reason": "startOffset must be non-negative, and endOffset must be \>= startOffset, and offsets must not go backwards startOffset=6,endOffset=17,lastStartOffset=14 for field 'content'"
> },
> "status": 400
> }
> \`\`\`
> 
> 4. Use the Analyze API to view the tokens:
> \`\`\`
> GET /test\_index/\_analyze
> {
> "analyzer" : "stemmed\_en",
> "text" : "email abc@def.com"
> }
> \`\`\`
> and the result is:
> \`\`\`
> {
> "tokens": \[
> {
> "token": "email",
> "start\_offset": 0,
> "end\_offset": 5,
> "type": "word",
> "position": 0
> },
> {
> "token": "e",
> "start\_offset": 0,
> "end\_offset": 5,
> "type": "SYNONYM",
> "position": 0
> },
> {
> "token": "abc",
> "start\_offset": 6,
> "end\_offset": 9,
> "type": "word",
> "position": 1
> },
> {
> "token": "def",
> "start\_offset": 10,
> "end\_offset": 13,
> "type": "word",
> "position": 2
> },
> {
> "token": "com",
> "start\_offset": 14,
> "end\_offset": 17,
> "type": "word",
> "position": 3
> },
> {
> "token": "mail",
> "start\_offset": 6,
> "end\_offset": 17,
> "type": "SYNONYM",
> "position": 3
> }
> \]
> }
> \`\`\`
> 
> \### Logs (if relevant)
> 
> \_No response\_

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2022, 8:51am UTC](https://discuss.elastic.co/t/tokens-offset-issue/307737/3 "2022-07-21T08:51:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.