# Elasticsearch - Limit overlaps in shingle token filter results - using Frank Ocean as a dataset

**URL:** <https://discuss.elastic.co/t/elasticsearch-limit-overlaps-in-shingle-token-filter-results-using-frank-ocean-as-a-dataset/43964>\
**Category:** Elasticsearch\
**Created:** [March 10, 2016, 12:21am UTC](https://discuss.elastic.co/t/elasticsearch-limit-overlaps-in-shingle-token-filter-results-using-frank-ocean-as-a-dataset/43964 "2016-03-10T00:21:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![unsalted](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/unsalted/32/8393_2.png) [@unsalted](https://discuss.elastic.co/u/unsalted)\
**Post date:** [March 10, 2016, 12:21am UTC](https://discuss.elastic.co/t/elasticsearch-limit-overlaps-in-shingle-token-filter-results-using-frank-ocean-as-a-dataset/43964/1 "2016-03-10T00:21:48Z")

</div>

Ok as a test dataset I have 100 or so lyrics from Frank Ocean - yes... Frank Ocean.

I am using this to build up my autocomplete filter:

```
PUT /test_songs
{
  "settings": {
    "index": {
      "analysis": {
        "filter": {
          "stemmer": {
            "type": "stemmer",
            "language": "english"
          },
          "autocompleteFilter": {
            "max_shingle_size": 10,
            "min_shingle_size": 4,
            "output_unigrams_if_no_shingles": true,
            "output_unigrams" : false,
            "type": "shingle"
          },
          "stopwords": {
            "type": "stop",
            "stopwords": [
              "_english_"
            ]
          }
        },
        "analyzer": {
          "didYouMean": {
            "filter": [
              "lowercase"
            ],
            "type": "custom",
            "tokenizer": "standard"
          },
          "autocomplete": {
            "filter": [
              "lowercase",
              "autocompleteFilter"
            ],
            "type": "custom",
            "tokenizer": "standard"
          },
          "default": {
            "filter": [
              "lowercase",
              "stopwords",
              "stemmer"
            ],
            "type": "custom",
            "tokenizer": "standard"
          }
        }
      }
    }
  }
}

```

Here is an example query:

```
GET /test_songs/_search/
{
  "size": 0,
  "aggs": {
    "autocomplete": {
      "terms": {
        "field": "autocomplete",
        "order": {
          "_count": "desc"
        },
        "include": {
          "pattern": "i want.*"
        }
      }
    }
  },
  "query": {
    "prefix": {
      "autocomplete": {
        "value": "i want"
      }
    }
  }
}

```

_And response_

```
"aggregations": {
    "autocomplete": {
      "doc_count_error_upper_bound": 1,
      "sum_other_doc_count": 158,
      "buckets": [
        {
          **"key": "i want to be",**
          "doc_count": 3
        },
        {
          "key": "i want to be the",
          "doc_count": 2
        },
        {
          **"key": "i want you to",**
          "doc_count": 2
        },
        {
          "key": "i want and baby",
          "doc_count": 1
        },
        {
          "key": "i want and baby i",
          "doc_count": 1
        },
        {
          "key": "i want and baby i ain't",
          "doc_count": 1
        },
        {
          "key": "i want and baby i ain't never",
          "doc_count": 1
        },
        {
          "key": "i want and baby i ain't never fell",
          "doc_count": 1
        },
        {
          "key": "i want and baby i ain't never fell in",
          "doc_count": 1
        },
        {
          **"key": "i want and baby i ain't never fell in love",**
          "doc_count": 1
        }
      ]
    }
  }

```

**I need more of:**

- "i want to be"
- "i want you to"
- "i want and baby i ain't never fell in love"

And less of the buildup to those results.

**Suggestions?**

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:09pm UTC](https://discuss.elastic.co/t/elasticsearch-limit-overlaps-in-shingle-token-filter-results-using-frank-ocean-as-a-dataset/43964/2 "2017-07-05T23:09:44Z")

</div>

Closed on July 5, 2017, 11:09pm UTC