Asciifolding doesn't exist anymore in 6.0.x?

obruchez · December 11, 2017, 2:45pm

I was using a custom analyzer with two filters: lowercase + asciifolding.

It doesn't work anymore in 6.0.0 and 6.0.1.

The easiest way to reproduce the problem:

POST /_analyze
{
  "tokenizer": "standard",
  "filter":  [ "lowercase", "asciifolding" ],
  "text":      "Is this déja vu?"
}

Result:

{
    "error": {
        "root_cause": [
            {
                "type": "remote_transport_exception",
                "reason": "[pyJpaTt][127.0.0.1:9300][indices:admin/analyze[s]]"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "failed to find global token filter under [asciifolding]"
    },
    "status": 400
}

If I remove the asciifolding filter:

{
    "tokens": [
        {
            "token": "is",
            "start_offset": 0,
            "end_offset": 2,
            "type": "<ALPHANUM>",
            "position": 0
        },
        {
            "token": "this",
            "start_offset": 3,
            "end_offset": 7,
            "type": "<ALPHANUM>",
            "position": 1
        },
        {
            "token": "déja",
            "start_offset": 8,
            "end_offset": 12,
            "type": "<ALPHANUM>",
            "position": 2
        },
        {
            "token": "vu",
            "start_offset": 13,
            "end_offset": 15,
            "type": "<ALPHANUM>",
            "position": 3
        }
    ]
}

Has asciifolding been removed? Is it available via a plugin? Should I use something else?

dadoonet · December 11, 2017, 3:01pm

Any chance you are not starting elasticsearch with bin/elasticsearch but by another method?

obruchez · December 11, 2017, 3:30pm

Yes, as a local node from a Java/Scala application. I guess it's not supported anymore, right?

dadoonet · December 11, 2017, 3:41pm

Running embedded is not supported indeed.