# Phrase suggester not giving suggestion when there are duplicate entries

**URL:** <https://discuss.elastic.co/t/phrase-suggester-not-giving-suggestion-when-there-are-duplicate-entries/97375>\
**Category:** Elasticsearch\
**Created:** [August 17, 2017, 9:29am UTC](https://discuss.elastic.co/t/phrase-suggester-not-giving-suggestion-when-there-are-duplicate-entries/97375 "2017-08-17T09:29:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![wmjb](https://avatars.discourse-cdn.com/v4/letter/w/76d3ee/32.png) [@wmjb](https://discuss.elastic.co/u/wmjb)\
**Post date:** [August 17, 2017, 9:29am UTC](https://discuss.elastic.co/t/phrase-suggester-not-giving-suggestion-when-there-are-duplicate-entries/97375/1 "2017-08-17T09:29:20Z")

</div>

Hi, I was playing with phrase suggester and was using sample code in the elastic search documention.

Mapping

```
PUT /test?pretty
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "analysis": {
        "analyzer": {
          "trigram": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": [
              "standard",
              "shingle"
            ]
          },
          "reverse": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": [
              "standard",
              "reverse"
            ]
          }
        },
        "filter": {
          "shingle": {
            "type": "shingle",
            "min_shingle_size": 2,
            "max_shingle_size": 3
          }
        }
      }
    }
  },
  "mappings": {
    "test": {
      "properties": {
        "title": {
          "type": "text",
          "fields": {
            "trigram": {
              "type": "text",
              "analyzer": "trigram"
            },
            "reverse": {
              "type": "text",
              "analyzer": "reverse"
            }
          }
        }
      }
    }
  }
}

```

Data

```
POST /test/test?refresh=true&pretty
{"title": "noble warriors"}
POST /test/test?refresh=true&pretty
{"title": "nobel prize"}

```

Search query

```
POST test/_search
{
  "suggest": {
    "text": "noble prize",
    "simple_phrase": {
      "phrase": {
        "field": "title.trigram",
        "size": 1,
        "gram_size": 3,
        "direct_generator": [ {
          "field": "title.trigram",
          "suggest_mode": "always"
        } ],
        "highlight": {
          "pre_tag": "<em>",
          "post_tag": "</em>"
        }
      }
    }
  }
}

```

Now above code works.  
But when I put more duplicate documents as follows phrase suggester stop giving any results.

```
POST /test/test?refresh=true&pretty
{"title": "noble warriors"}
POST /test/test?refresh=true&pretty
{"title": "nobel prize"}

POST /test/test?refresh=true&pretty
{"title": "noble warriors"}
POST /test/test?refresh=true&pretty
{"title": "nobel prize"}

```

Any idea why this is happening?

Thank you,

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [August 17, 2017, 2:59pm UTC](https://discuss.elastic.co/t/phrase-suggester-not-giving-suggestion-when-there-are-duplicate-entries/97375/2 "2017-08-17T14:59:10Z")

</div>

I think what you're running into here is that common (high frequency) words are excluded from suggestions by default. By indexing the same text over and over again, those terms become very common and excluded from suggestions.

If you really want those terms to be suggested, you could add the `max_term_freq` parameter to `direct_generator` with a very high fraction (for example `0.999`). Your example would become:

```
POST test/_search
{
  "suggest": {
    "text": "noble prize",
    "simple_phrase": {
      "phrase": {
        "field": "title.trigram",
        "size": 1,
        "gram_size": 3,
        "direct_generator": [ {
          "field": "title.trigram",
          "max_term_freq": 0.999,
          "suggest_mode": "always"
        } ],
        "highlight": {
          "pre_tag": "<em>",
          "post_tag": "</em>"
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![wmjb](https://avatars.discourse-cdn.com/v4/letter/w/76d3ee/32.png) [@wmjb](https://discuss.elastic.co/u/wmjb)\
**Post date:** [August 18, 2017, 8:10am UTC](https://discuss.elastic.co/t/phrase-suggester-not-giving-suggestion-when-there-are-duplicate-entries/97375/3 "2017-08-18T08:10:53Z")

</div>

Thank you @abdon for a perfect explanation. 👌

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2017, 8:10am UTC](https://discuss.elastic.co/t/phrase-suggester-not-giving-suggestion-when-there-are-duplicate-entries/97375/4 "2017-09-15T08:10:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.