Synyonyms get applied to split tokens from word_delimiter_graph with multiplexer?

[olucafont6]

If you set up an index with token filters like this:

PUT /test_index

POST /test_index/_close

PUT /test_index/_settings
{
  "analysis": {
    "filter": {
      "synonyms": {
        "type": "synonym",        
        "synonyms": [
          "one => synonym"
        ]
      },
      "word_splitter": {
        "type": "word_delimiter_graph",
        "preserve_original": true
      }
    }
  }
}

POST /test_index/_open

And then make a request like this:

POST test_index/_analyze
{
  "tokenizer": "whitespace",
  "filter": [
    {
      "type": "multiplexer",
      "filters": [
        "word_splitter, flatten_graph",
        "synonyms"
      ]
    }
  ],
  "text": "one/two"
}

The synonym for "one" will get applied to the split token ("one/two" => "one", "two"):

{
  "tokens": [
    {
      "token": "one/two",
      "start_offset": 0,
      "end_offset": 7,
      "type": "word",
      "position": 0
    },
    {
      "token": "synonym",
      "start_offset": 0,
      "end_offset": 3,
      "type": "SYNONYM",
      "position": 1
    },
    {
      "token": "two",
      "start_offset": 4,
      "end_offset": 7,
      "type": "word",
      "position": 2
    },
    {
      "token": "one/two",
      "start_offset": 0,
      "end_offset": 7,
      "type": "word",
      "position": 2
    }
  ]
}

I don't understand why this works, since the multiplexer documentation doesn't say anything about the token filter lists interacting.

Though it does say:

Shingle or multi-word synonym token filters will not function normally when they are declared in the filters array because they read ahead internally which is unsupported by the multiplexer

Options - Multiplexer token filter | Elasticsearch Reference

And then later:

The synonym and synonym_graph filters use their preceding analysis chain to parse and analyse their synonym lists, and will throw an exception if that chain contains token filters that produce multiple tokens at the same position. If you want to apply synonyms to a token stream containing a multiplexer, then you should append the synonym filter to each relevant multiplexer filter list, rather than placing it after the multiplexer in the main token chain definition.

Settings example - Multiplexer token filter | Elasticsearch Reference

So maybe this is just some undefined behavior, and this just happens to work?

Interestingly if I make any of these changes, it stops working (the synonym for "one" doesn't get applied):

{
  "word_splitter": {
    "type": "word_delimiter_graph",
    "preserve_original": false
  }
}

But interestingly if I alternatively enable this setting, it also works:

{
  "word_splitter": {
    "type": "word_delimiter_graph",
    "catenate_words": true
  }
}

Also weirdly this doesn't work:

{
  "synonyms": {
    "type": "synonym",
    "synonyms": [
      "one, synonym"
    ]
  }
}

But this does:

{
  "synonyms": {
    "type": "synonym",
    "synonyms": [
      "one, synonym"
    ],
    "catenate_words": false
  }
}

And also if I remove the flatten_graph filter from after the word_splitter filter, it stops working:

POST test_index/_analyze
{
  "tokenizer": "whitespace",
  "filter": [
    {
      "type": "multiplexer",
      "filters": [
        "word_splitter",
        "synonyms"
      ]
    }
  ],
  "text": "one/two"
}

Since this seems extremely brittle (you have to have three separate things configured in a certain way, and it doesn't really make sense why those would lead to this result), I'm guessing this is just a bug.

The thing is, I'm looking for a way to achieve this result (split tokens with a word_delimiter_graph and then apply synonyms to the results), but there doesn't seem to actually be a supported way of doing this, re this guy's question:

Word Delimiter Graph Token + Synonym Graph Token

I want to use a word delimiter graph token to avoid spliting terms with "+" and "-", e.g. so "3+c" doesn't change to ["3","c"] and stays ["3+c"]. I also want to define a list of synonyms to be applied to this field after filter tokenization. Documentation says: If you need to build analyzers that include both multi-token filters and synonym filters, consider using the multiplexer filter, with the multi-token filters in one branch and the synonym filter in the other. but I don't see how my…

I didn't really even want to use the multiplexer in the first place, it's just that if I try directly applying the synonym filter after the word_delimiter_graph, I get the "Token filter [word_splitter] cannot be used to parse synonyms" error.

So I'm not sure how to proceed - do I just rely on this bug / undefined behavior, or is there a cleaner way of doing this?

[willsun]

Yes, this happens because multiplexer runs all of its child filter chains on the same incoming token stream in parallel and emits the union of tokens at the same positions, unless you separate them between index time and search time.

Looking closely at your _analyze call:

"filter": [
  {
    "type": "multiplexer",
    "filters": [
      "word_splitter, flatten_graph",
      "synonyms"
    ]
  }
]

If synonyms is a separate branch inside multiplexer, it only sees the unsplit token from whitespace unless word_delimiter_graph with preserve_original: true is emitting overlapping position tokens or your test input already had "one" separated by whitespace.

In practice, mixing word_delimiter_graph + synonym_graph at index time almost always creates weird position-length side effects for phrase queries. On our product catalog and site-search indices (my profile has links to our search & GEO case notes), we keep word_delimiter_graph + flatten_graph strictly in the index analyzer, and run synonym_graph strictly in the search analyzer without multiplexer. What exact input string did you pass to _analyze when you saw the split token get synonym-expanded?

[olucafont6]

@willsun

If synonyms is a separate branch inside multiplexer, it only sees the unsplit token from whitespace unless word_delimiter_graph with preserve_original: true is emitting overlapping position tokens or your test input already had "one" separated by whitespace.

Okay, I'm still not sure why preserve_original would cross the streams like this, but maybe this is undefined (or at least undocumented behavior), re your following point:

@willsun

In practice, mixing word_delimiter_graph + synonym_graph at index time almost always creates weird position-length side effects for phrase queries.

@willsun

On our product catalog and site-search indices (my profile has links to our search & GEO case notes)

Hmm, it says your public profile is hidden? I would be curious to see that though.

@willsun

we keep word_delimiter_graph + flatten_graph strictly in the index analyzer, and run synonym_graph strictly in the search analyzer without multiplexer.

Yeah, this is what I originally wanted to do (apply the synonyms through the search analyzer), but there was an aspect of the synonym behavior that didn't work with this setup. Basically we'd have synonym entries like "one, multi word", and then we'd want to be able to match "multi" or "word" in the search query on an indexed field like "one". Basically we'd want to be able to match on just one of the multi-word synonym words, and have that count as a full synonym match (e.g., "multi => one", "word => one"). But maybe we just need to rework our synonym lists so these expectations are more explicit...

@willsun

What exact input string did you pass to _analyze when you saw the split token get synonym-expanded?

It's the second code snippet in my original post, but it was "one/two".