Match_phrase_prefix on keyword field inconsistent behavior between ES 5.6.8 and ES 6+

Elastic Stack Elasticsearch
1

I'm noticing different behavior on Elasticsearch 5.6.8 and Elasticsearch 6+ on doing a match_phrase_prefix on keyword. I was able to do a match_phrase_prefix on a keyword exact match on Elasticsearch 5.6.8 but not on Elasticsearch 6.

I suspect it's because of the lowercase normalizer not being applied upon query in ES 6. Issuing a match_phrase_prefix in lowercase returns the document, but doesn't return anything if issued an uppercase

Step to reproduce:

// Create a an index
curl -X PUT 'localhost:9200/my_index_6'

// Create index mapping
    curl -X PUT 'localhost:9200/my_index_6/records/_mapping?pretty' -H 'Content-Type: application/json' -d '
    {
        "records": {
            "dynamic": "strict",
            "properties": {
                "my_field": {
                    "type": "keyword",
                    "fields": {
                        "caseinsensitive": {
                            "type": "keyword",
                            "ignore_above": 10922,
                            "normalizer": "lowercaseanalyzer"
                        }
                    }
                }
            }
        }
    }'

 // Add settings
    curl -X PUT 'localhost:9200/my_index_6' -H 'Content-Type: application/json' -d '
    {
        "settings": {
            "index": {
              "analysis": {
                "filter": {
                  "english_stopwords": {
                    "type": "stop",
                    "stopwords_path": "stopwords/stopwords_all.txt"
                  },
                  "english_stemmer": {
                    "name": "light_english",
                    "type": "stemmer"
                  },
                  "english_possessive_stemmer": {
                    "name": "possessive_english",
                    "type": "stemmer"
                  },
                  "english_stemmer_override": {
                    "type": "stemmer_override",
                    "rules_path": "stemmer_override/stemmer_override_english.txt"
                  }
                },
                "normalizer": {
                  "lowercaseanalyzer": {
                    "filter": [
                      "lowercase"
                    ],
                    "type": "custom"
                  }
                },
                "analyzer": {
                  "english": {
                    "filter": [
                      "english_possessive_stemmer",
                      "lowercase",
                      "english_stopwords",
                      "english_stemmer_override",
                      "english_stemmer"
                    ],
                    "type": "custom",
                    "tokenizer": "standard"
                  }
                }
              }
            }
          }    
    }
    '

// Create a document
curl -X POST 'localhost:9200/my_index_6/records' -H 'Content-Type: application/json' -d '
{
    "my_field": "ID01"
}
'

// Issue a search query. This WORKS on ES 5 but DOESN'T WORK on ES 6
curl -X GET "localhost:9200/my_index_6/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "match_phrase_prefix" : {
            "my_field.caseinsensitive" : {
                "query" : "ID01",
                "boost" : 1.0
            }
        }
    }
}
'

// THIS WORKS 
    curl -X GET "localhost:9200/my_index_6/_search?pretty" -H 'Content-Type: application/json' -d'
    {
        "query": {
            "match_phrase_prefix" : {
                "my_field.caseinsensitive" : {
                    "query" : "id01",
                    "boost" : 1.0
                }
            }
        }
    }
    '

Appreciate any help on this.

Thanks

2

@warkolm thoughts on this? In case you are not the right person, can you please redirect this to someone who can help?