# Highlighting Offset Issue with Vietnamese

**URL:** <https://discuss.elastic.co/t/highlighting-offset-issue-with-vietnamese/50421>\
**Category:** Elasticsearch\
**Created:** [May 19, 2016, 10:22am UTC](https://discuss.elastic.co/t/highlighting-offset-issue-with-vietnamese/50421 "2016-05-19T10:22:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![drippel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drippel/32/44823_2.png) [@drippel](https://discuss.elastic.co/u/drippel)\
**Post date:** [May 19, 2016, 10:22am UTC](https://discuss.elastic.co/t/highlighting-offset-issue-with-vietnamese/50421/1 "2016-05-19T10:22:37Z")

</div>

We are using ElasticSearch 2.2.0 with the compatible [Vietnamese Plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese).  
When I run the following query:

```
{
  "query": {
    "bool": {
      "should": {
        "multi_match": {
          "query": "chữ",
          "fields": [
            "message",
            "user"
          ],
          "analyzer": "default_search"
        }
      }
    }
  },
  "highlight": {
    "pre_tags": [
      "<b>"
    ],
    "post_tags": [
      "</b>"
    ],
    "fragment_size": 0,
    "number_of_fragments": 0,
    "require_field_match": false,
    "fields": {
      "message": {},
      "user": {}
    }
  }
}

```

I get the following result, where the highlighting offsets are wrong and the tags wrap the wrong words, although the actual search did work and retrieved the right document:

```
{
    "took" : 14,
    "timed_out" : false,
    "_shards" : {
        "total" : 200,
        "successful" : 200,
        "failed" : 0
    },
    "hits" : {
        "total" : 1,
        "max_score" : 0.030578919,
        "hits" : [{
                "_index" : "fts-vietnamese",
                "_type" : "Document",
                "_id" : "AVKcb6Xy0-uCokJzleqC",
                "_score" : 0.030578919,
                "_source" : {
                    "streamId" : 1,
                    "language" : "vietnamese",
                    "message" : "Có một vấn đề là khi sent text messages dùng tiếng Việt hoặc email qua người khác, chữ tiếng Việt bị mất dấu hoặc mất chữ. Chẳng hạn như chữ “ôm” thì thành",
                    "doc_id" : "VietnameseWords"
                },
                "highlight" : {
                    "message" : [
                        "Có một vấn đề là khi sent text messages dùng tiếng Việt hoặc email <b>qua</b> người khác, chữ tiếng V<b>iệt</b> bị mất dấu h<b>oặc</b> mất chữ. Chẳng hạn như chữ “ôm” thì thành"
                    ]
                }
            }
        ]
    }
}

```

@Duy_Do who created the Vietnamese plugin thinks that the problem is in the ES highlight parser that is not working properly with Vietnamese characters.  
Does anyone have a solution to this?  
Thanks!

---

<div class="post-metadata">

**Author:** ![Andrei\_Stefan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei_stefan/32/47533_2.png) [@Andrei\_Stefan](https://discuss.elastic.co/u/Andrei_Stefan)\
**Post date:** [May 24, 2016, 12:12pm UTC](https://discuss.elastic.co/t/highlighting-offset-issue-with-vietnamese/50421/2 "2016-05-24T12:12:18Z")

</div>

The analyzer should do the correct calculation of the offsets of each term.  
In this particular case, using `GET /index/type/doc_id/_termvectors?pretty=true&fields=message` to see the offsets, they are incorrect:

```auto
            "chữ": {
               "term_freq": 3,
               "tokens": [
                  {
                     "position": 17,
                     "start_offset": 67,
                     "end_offset": 70
                  },
                  {
                     "position": 33,
                     "start_offset": 94,
                     "end_offset": 97
                  },
                  {
                     "position": 42,
                     "start_offset": 110,
                     "end_offset": 113
                  }
               ]
            }

```

And this is the reason why the highlighting fails. So, the analyzer/tokenizer should properly take care of the offsets.

---

<div class="post-metadata">

**Author:** ![drippel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drippel/32/44823_2.png) [@drippel](https://discuss.elastic.co/u/drippel)\
**Post date:** [May 24, 2016, 12:27pm UTC](https://discuss.elastic.co/t/highlighting-offset-issue-with-vietnamese/50421/3 "2016-05-24T12:27:12Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:49pm UTC](https://discuss.elastic.co/t/highlighting-offset-issue-with-vietnamese/50421/4 "2017-07-05T22:49:32Z")

</div>