# Kuromoji token offsets going backwards in extended mode

**URL:** <https://discuss.elastic.co/t/kuromoji-token-offsets-going-backwards-in-extended-mode/253634>\
**Category:** Elasticsearch\
**Created:** [October 29, 2020, 2:49am UTC](https://discuss.elastic.co/t/kuromoji-token-offsets-going-backwards-in-extended-mode/253634 "2020-10-29T02:49:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![matobaa](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@matobaa](https://discuss.elastic.co/u/matobaa)\
**Post date:** [October 29, 2020, 2:49am UTC](https://discuss.elastic.co/t/kuromoji-token-offsets-going-backwards-in-extended-mode/253634/1 "2020-10-29T02:49:06Z")

</div>

When we upgraded our ES instance from 6.8 to 7.8, we noticed errors during indexing like below:

java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be \>= startOffset, and offsets must not go backwards startOffset=178,endOffset  
=193,lastStartOffset=187 for field

When we examined the text, we noticed that the Kuromoji tokenizer in extended mode was producing tokens that had start offsets that went backwards:

```auto
{
	"tokenizer": {
		"type": "kuromoji_tokenizer",
		"mode": "extended"
	},
	"text": "株式会社ワーナーミュージック・ジャパン"
}

```

```auto
{
  "tokens": [
    {
      "token": "株式",
      "start_offset": 0,
      "end_offset": 2,
      "type": "word",
      "position": 0
    },
   ...
    {
      "token": "ク",
      "start_offset": 13,
      "end_offset": 14,
      "type": "word",
      "position": 11
    },
    {
      "token": "ワーナーミュージック・ジャパン",
      "start_offset": 4,
      "end_offset": 19,
      "type": "word",
      "position": 12
    },
  
  ]
}

```

This appears to be a bug. Are there any ways to work around this issue? Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 26, 2020, 2:49am UTC](https://discuss.elastic.co/t/kuromoji-token-offsets-going-backwards-in-extended-mode/253634/2 "2020-11-26T02:49:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.