# Ignore shingle tokens when highlighting

**URL:** <https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695>\
**Category:** Elasticsearch\
**Created:** [January 16, 2018, 10:54am UTC](https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695 "2018-01-16T10:54:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bslinger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bslinger/32/26610_2.png) [@bslinger](https://discuss.elastic.co/u/bslinger)\
**Post date:** [January 16, 2018, 10:54am UTC](https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695/1 "2018-01-16T10:54:18Z")

</div>

I'm using shingles in our index to allow for matches on multi-words when a search using a joined version of the word is entered, eg. to allow 'giantbomb' to match a field with the term 'giant bomb' in it.

This is working fine, but because I shingle the whole field, the highlighter is matching the shingles and highlighting additional words. Is there a way to tell the highlighter to only match on tokens that aren't generated by the shingle filter?

Example:

`DELETE /test`

```
PUT /test
{
  "settings": {
    "index": {
      "number_of_replicas": 0,
      "analysis": {
          "filter": {
            "shingle_join": {
              "max_shingle_size": "2",
              "token_separator": "",
              "output_unigrams": "true",
              "type": "shingle",
              "min_shinge_size": "2"
            },
            "autocomplete_filter": {
              "type": "edge_ngram",
              "min_gram": "1",
              "max_gram": "20"
            }
          },
          "analyzer": {
            "autocomplete": {
              "filter": [
                "lowercase",
                "shingle_join",
                "autocomplete_filter"
              ],
              "type": "custom",
              "tokenizer": "standard"
            }
          }
      }
    }
  },
  "mappings": {
    "test":{
      "properties": {
        "test":{
          "type":"string",
          "analyzer": "autocomplete"
        }
      }
    }
  }
}

PUT /test/test/1
{
  "test":"giant bomb"
}

GET /test/test/_search
{
  "query":{
    "match":{
      "test":{
        "query":"giant"
      }
    }
  },
  "highlight": {
    "fields": {
      "test":{}
    }
  }
}

```

Output:

```
    {
      "took": 16334,
      "timed_out": false,
      "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
      },
      "hits": {
        "total": 1,
        "max_score": 0.60692424,
        "hits": [
          {
            "_index": "test",
            "_type": "test",
            "_id": "1",
            "_score": 0.60692424,
            "_source": {
              "test": "giant bomb"
            },
            "highlight": {
              "test": [
                "<em>giant bomb</em>"
              ]
            }
          }
        ]
      }
    }

```

Desired output:

```
{
  "took": 16334,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.60692424,
    "hits": [
      {
        "_index": "test",
        "_type": "test",
        "_id": "1",
        "_score": 0.60692424,
        "_source": {
          "test": "giant bomb"
        },
        "highlight": {
          "test": [
            "<em>giant</em> bomb"
          ]
        }
      }
    ]
  }
}
```

---

<div class="post-metadata">

**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\
**Post date:** [January 19, 2018, 1:31pm UTC](https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695/2 "2018-01-19T13:31:58Z")

</div>

did you try a `highlight_query` (see [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#highlighting-settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#highlighting-settings) that might help here. If you run in to issues please ping back. You might also need to set `min_shingle_size` to `1` otherwise it won't match a single value

---

<div class="post-metadata">

**Author:** ![bslinger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bslinger/32/26610_2.png) [@bslinger](https://discuss.elastic.co/u/bslinger)\
**Post date:** [January 20, 2018, 12:08pm UTC](https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695/3 "2018-01-20T12:08:49Z")

</div>

Hi Simon,

I get tokens for the single words because I'm using "output\_unigrams":"true", right? Here are the tokens for that term using that analyzer:

```
{
  "tokens": [
    {
      "token": "g",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "gi",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "gia",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "gian",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "giant",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "g",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "gi",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "gia",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "gian",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "giant",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "giantb",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "giantbo",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "giantbom",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "giantbomb",
      "start_offset": 0,
      "end_offset": 10,
      "type": "shingle",
      "position": 0,
      "positionLength": 2
    },
    {
      "token": "b",
      "start_offset": 6,
      "end_offset": 10,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "bo",
      "start_offset": 6,
      "end_offset": 10,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "bom",
      "start_offset": 6,
      "end_offset": 10,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "bomb",
      "start_offset": 6,
      "end_offset": 10,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}

```

I did try the highlight\_query but I'm not sure how to get it to ignore certain tokens - trying the following gets me the same highlight results as in my original post:

```
GET /test/test/_search
{
  "query":{
    "match":{
      "test":{
        "query":"giant"
      }
    }
  },
  "highlight": {
    "fields": {
      "test":{}
    },
    "highlight_query":{
      "match":{
        "test":{
          "query":"giant"
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![bslinger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bslinger/32/26610_2.png) [@bslinger](https://discuss.elastic.co/u/bslinger)\
**Post date:** [January 20, 2018, 10:29pm UTC](https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695/4 "2018-01-20T22:29:02Z")

</div>

Just to add to this, I tried created a separate field that isn't using shingles so I could have the highlight\_query use that instead - this gave desired results in certain situations, but not others.

Here are my new mappings:

```
PUT /test
{
  "settings": {
    "index": {
      "number_of_replicas": 0,
      "analysis": {
          "filter": {
            "shingle_join": {
              "max_shingle_size": "2",
              "token_separator": "",
              "output_unigrams": "true",
              "type": "shingle",
              "min_shingle_size": "2"
            },
            "autocomplete_filter": {
              "type": "edge_ngram",
              "min_gram": "1",
              "max_gram": "20"
            }
          },
          "analyzer": {
            "autocomplete": {
              "filter": [
                "lowercase",
                "autocomplete_filter"
              ],
              "type": "custom",
              "tokenizer": "standard"
            },
            "shingle": {
              "filter": [
                "lowercase",
                "shingle_join"
              ],
              "type": "custom",
              "tokenizer": "standard"
            }
          }
      }
    }
  },
  "mappings": {
    "test":{
      "properties": {
        "test":{
          "type":"string",
          "analyzer": "shingle",
          "fields":{
            "autocomplete": {
              "type":"string",
              "analyzer": "autocomplete"
            }
          }
        }
      }
    }
  }
}

```

And my new query:

```
GET /test/test/_search
{
  "query":{
    "match":{
      "test.autocomplete":{
        "query":"giantbo"
      }
    }
  },
  "highlight": {
    "fields": {
      "test":{}
    },
    "highlight_query":{
      "match":{
        "test":{
          "query":"giantbo"
        }
      }
    }
  }
}

```

This works correctly if I don't require it to match on the nGrams created by the "autocomplete" analyzer - search terms for both "giant" and "giant bomb" and "giantbomb" all work correctly, but searching for "giantbo" as above matches the main query but not the highlight query.

---

<div class="post-metadata">

**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\
**Post date:** [January 22, 2018, 9:18am UTC](https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695/5 "2018-01-22T09:18:24Z")

</div>

I think the problem in your case is due to the fact that we have to preserve the offsets during indexing and if you choose a different indexing strategy stuff like the compounding / de-compounding won't really work. I think you need have to make a decision with a certain tradeoff. all the possible highlights that you are looking for won't work.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 19, 2018, 9:18am UTC](https://discuss.elastic.co/t/ignore-shingle-tokens-when-highlighting/115695/6 "2018-02-19T09:18:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.