Case Insensitive aggregation not working

[raanup (Anup)]

ES version - 7.17.7

I've an index for which I' running an aggregation to get all field matching certain regex. This should be case-insensitive i.e.

new york should match New York and NEW YORK and New YORK

Have added a lowercase_normaliser to index so that documents are indexed with lowercase name. This doesn't solve the issue though.
Now I've to pass the regex in lowercase, else it doesn't return correct results.

I've created an index named blah with following mapping

{
  "blah": {
    "mappings": {
      "properties": {
        "name": {
          "type": "text",
          "fields": {
            "raw": {
              "type": "keyword",
              "normalizer": "lowercase_normalizer"
            }
          }
        }
      }
    }
  }
}

Index settings -

{
  "blah": {
    "settings": {
      "index": {
        "max_ngram_diff": "20",
        "analysis": {
          "normalizer": {
            "lowercase_normalizer": {
              "filter": [
                "lowercase"
              ],
              "type": "custom"
            }
          }
        }
      }
    }
  }
}

Documents inserted:

POST blah/_doc/1

{
  "name": "NEW YORK"
}

POST blah/_doc/2

{
  "name": "New Orleans"
}

POST blah/_doc/3

{
  "name": "New Hampshire"
}

Following aggregation query doesn't return expected results -
Query

{
  "aggregations": {
    "autoComplete": {
      "terms": {
        "field": "name.raw",
        "include": "New.*",
        "order": [
          {
            "_term": "asc"
          }
        ]
      }
    }
  },
  "size": 0
}

Output

  "aggregations": {
    "autoComplete": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [

      ]
    }
  }

Another downside is that if I use aggregation after running .lowecase on inlcude value, results are also normalised. Is it possible to have original value being returned?

EDIT
is there another way to get all possible values of a field, in this case, name other than aggregation?

[RabBit_BR (andre.coelho)]

Hi @raanup

You are using keyword type. Try search with term applying lowecase like this:

@raanup

"include": "new.*",

[raanup (Anup)]

I understand that. Apologies, haven't updated it. I'm actually using
"include": "new.*"
and I do get the hits but all of them are normalized.

[RabBit_BR (andre.coelho)]

I noticed that you want to do an autocomplete. Have you already researched other options like search_as_you_type or completion suggester?

[raanup (Anup)]

Thanks @RabBit_BR . Will try this and let you know