Highlight does not work when `store` is set to false and search value does not match normalized filter

[naoko]

ES version: 5.5.2

I've created index like this:

{
	"settings": {
		"index": {
		"analysis": {
			"normalizer": {
				"normalizer_lower": {
					"type": "custom",
					"filter": ["lowercase"]
				},
				"normalizer_upper": {
					"type": "custom",
					"filter": ["uppercase"]
				}
			}
		}}
	},
	"mappings": {
		"t": {
			"properties": {
				"kw_upper": {
					"type": "keyword",
					"normalizer": "normalizer_upper",
					"ignore_above": 256,
					"store": false
				},
				"kw_lower": {
					"type": "keyword",
					"normalizer": "normalizer_lower",
					"ignore_above": 256,
					"store": false
				},"kw_lower_store": {
					"type": "keyword",
					"normalizer": "normalizer_lower",
					"ignore_above": 256,
					"store": true
				}
			}
		}
	}
}

and put documents like these:

{ "index" : {"_id" : "1", "_type" : "t", "_index" : "test"}}
 {"kw_lower": "lower.123", "kw_upper": "lower.123", "kw_lower_store": "lower.123"}
{ "index" : {"_id" : "2", "_type" : "t", "_index" : "test"}}
 {"kw_lower": "UPPER.123", "kw_upper": "UPPER.123", "kw_lower_store": "UPPER.123"}

when I search "uppercase" normalized field with uppercased value ("UPPER.123")
it highlights kw_upper and kw_lower_store field only but not kw_lower.

when I search "lowercase" normalized field with lowercased value ("lower.123")
it highlights kw_lower and kw_lower_store field only but not kw_upper.

Is this by design? If so where may I find that in documentation?

Unless I misunderstand this note in doc. Doc states:

In order to perform highlighting, the actual content of the field is required. If the field in question is stored (has store set to true in the mapping) it will be used, otherwise, the actual _source will be loaded and the relevant field will be extracted from it.

the value is stored in _source so I expect that it works without store=true.

[johtani (Jun Ohtani)]

Can you share your search query?

[naoko]

Yes,

{
	"query": {
		"multi_match": {
			"fields": ["kw_upper", "kw_lower", "kw_lower_store"],
			"type": "most_fields",
			"query": "lower.123",
			"operator": "or"
		}
	},
	"highlight": {
		"fields": {
			"*": {}
		}
	}
}

[johtani (Jun Ohtani)]

Thx for sharing.

Maybe that was a bug related this PR.

In 6.2.0, you can get the following response:

GET hoge_highlight/_search
{
	"query": {
		"multi_match": {
			"fields": ["kw_upper", "kw_lower", "kw_lower_store"],
			"type": "most_fields",
			"query": "upper.123",
			"operator": "or"
		}
	},
	"highlight": {
		"fields": {
			"*": {}
		}
	}
}

{
  "took": 497,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.8630463,
    "hits": [
      {
        "_index": "hoge_highlight",
        "_type": "t",
        "_id": "2",
        "_score": 0.8630463,
        "_source": {
          "kw_lower": "UPPER.123",
          "kw_upper": "UPPER.123",
          "kw_lower_store": "UPPER.123"
        },
        "highlight": {
          "kw_upper": [
            "<em>UPPER.123</em>"
          ],
          "kw_lower": [
            "<em>UPPER.123</em>"
          ],
          "kw_lower_store": [
            "<em>upper.123</em>"
          ]
        }
      }
    ]
  }
}

[naoko]

Thank you very much! You nailed it. I confirmed this on ver: 6.2.0. We will upgrade to 6.2.0 to solve the issue.