# Synonym token graphs and Shingles don't play well together

**URL:** <https://discuss.elastic.co/t/synonym-token-graphs-and-shingles-dont-play-well-together/230713>\
**Category:** Elasticsearch\
**Created:** [May 1, 2020, 1:18pm UTC](https://discuss.elastic.co/t/synonym-token-graphs-and-shingles-dont-play-well-together/230713 "2020-05-01T13:18:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Silas\_Hansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silas_hansen/32/67521_2.png) [@Silas\_Hansen](https://discuss.elastic.co/u/Silas_Hansen)\
**Post date:** [May 1, 2020, 1:18pm UTC](https://discuss.elastic.co/t/synonym-token-graphs-and-shingles-dont-play-well-together/230713/1 "2020-05-01T13:18:01Z")

</div>

Hi

I know I'm not the first one to touch on this issue, but no solution seems to have been provided.

**Expected behavior:**  
With the synonym [world,earth] and the phrase "hello world", sending them through a shingle filter of size 2, I would expect the following terms to be produced.

```auto
"hello world"
"hello earth"

```

since "world" and "earth" share the same position due to the graph nature of the synonyms filter.

**Actual behavior:**  
However, it seems the Shingle filter is not looking at the position (graph) information of the terms produced by the synonyms\_graph filter, but instead just sees it as a flat list of [hello, earth, world], outputting the shingles:

```auto
"hello earth"
"earth world"

```

**Why should it work now?**  
I've read that the [index\_phrase](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-phrases.html) feature uses a lucene filter called [FixedShingleFilter](https://issues.apache.org/jira/browse/LUCENE-8202) to get around this limitation.

The _index\_phrase_ feature does not cover my needs for bigram-matching, so I'd like to be able to roll my own analyzer.

I'm currently testing with ES 7.6.2, running Lucene 8.4.0.  
The [FixedShingleFilter](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-phrases.html) issue shows that the functionality was added back in Lucene 7.4.

**So my question is:**  
can I access the _FixedShingleFilter_ as well from settings? Or am I missing another obvious solution?

Thanks in advance!

  

**Code for reference:**

```auto
PUT synonyms_test
{
  "settings": {
    "index": {
      "analysis": {
        "filter": {
          "bigram_filter": {
            "max_shingle_size": "2",
            "min_shingle_size": "2",
            "token_separator": " ",
            "output_unigrams": "false",
            "type": "shingle"
          },
          "synonym_graph": {
            "type": "synonym_graph",
            "synonyms": [
              "world,earth"
            ]
          }
        },
        "analyzer": {
          "bigrams": {
            "filter": [
              "synonym_graph",
              "lowercase",
              "bigram_filter"
            ],
            "type": "custom",
            "tokenizer": "standard"
          }
        }
      }
    }
  }
}

GET synonyms_test/_analyze
{
  "analyzer": "bigrams", 
  "text": "hello world"
}

```

Output:

```auto
{
  "tokens" : [
    {
      "token" : "hello earth",
      "start_offset" : 0,
      "end_offset" : 11,
      "type" : "shingle",
      "position" : 0
    },
    {
      "token" : "earth world",
      "start_offset" : 6,
      "end_offset" : 11,
      "type" : "shingle",
      "position" : 1
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Silas\_Hansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silas_hansen/32/67521_2.png) [@Silas\_Hansen](https://discuss.elastic.co/u/Silas_Hansen)\
**Post date:** [May 7, 2020, 12:41pm UTC](https://discuss.elastic.co/t/synonym-token-graphs-and-shingles-dont-play-well-together/230713/2 "2020-05-07T12:41:34Z")

</div>

Are there really no one who can answer or provide an update on this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 4, 2020, 12:41pm UTC](https://discuss.elastic.co/t/synonym-token-graphs-and-shingles-dont-play-well-together/230713/3 "2020-06-04T12:41:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.