Issue with Kibana sorting- Caused by: SearchParseException nested: IllegalStateException[Field data loading is forbidden on severity]

[jenny8]

Elasticsearch, Logstash and Kibana are running.

kibana-4.3.1
elasticsearch-2.1.1

But failed to sort on any field from Kabana UI, for example, sort on field “severity”, got exception in elastic search.log.

I read the whole thread https://github.com/elastic/elasticsearch/issues/15267, it said it is not a bug, it is by design,that kibana does not allow to sort on analyzed fields. But I still do not get idea how to fix my Kabana sorting issue?

We should use severity.raw, which does exist in index pattern page, we can use severity.raw to create visualization, no problem. But severity.raw does not appear on Discover left panel, only severity. If I change the configuration of Available Fields with Analyzed=no, severity.raw is not in the list.

[2018-02-26 17:06:27,748][DEBUG][action.search.type ] [ACP10] All shards failed for phase: [query]

RemoteTransportException[[ACP10][10.10.44.43:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647},"query":{"filtered":{"query":{"query_string":{"query":"type:AcpLog","analyze_wildcard":true}},"filter":{"bool":{"must":[{"query":{"query_string":{"query":"","analyze_wildcard":true}}},{"range":{"@timestamp":{"gte":1361916387285,"lte":1519682787285,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":500,"sort":[{"severity":{"order":"asc","unmapped_type":"boolean"}}],"fields":["","_source"],"script_fields":{},"fielddata_fields":["@timestamp","arrivalTime"]}]]; nested: IllegalStateException[Field data loading is forbidden on severity];

Caused by: SearchParseException[failed to parse search source [{"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647},"query":{"filtered":{"query":{"query_string":{"query":"type:AcpLog","analyze_wildcard":true}},"filter":{"bool":{"must":[{"query":{"query_string":{"query":"","analyze_wildcard":true}}},{"range":{"@timestamp":{"gte":1361916387285,"lte":1519682787285,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":500,"sort":[{"severity":{"order":"asc","unmapped_type":"boolean"}}],"fields":["","_source"],"script_fields":{},"fielddata_fields":["@timestamp","arrivalTime"]}]]; nested: IllegalStateException[Field data loading is forbidden on severity];

[tsullivan (Tim Sullivan)]

If you have severity and severity.raw, then it sounds like severity is mapped as string, but is a multi field that has a "raw" field for the non-analyzed representation. Here's the documentation on multi-field: https://www.elastic.co/guide/en/elasticsearch/reference/2.1/multi-fields.html

You can change the mapping so severity becomes the "raw" and in the multi-field, the analyzed representation is severity.analyzed.

PUT /my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "severity": {
          "type": "string",
          "index": "not_analyzed",
          "fields": {
            "analyzed": { 
              "type":  "string"
            }
          }
        }
      }
    }
  }
}

More to the point though, do you actually need severity to be a multi-field with an analyzed representation, or can severity just be mapped as not_analyzed?

PUT /my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "severity": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  }
}

[jenny8]

Tim, thanks for your reply.

I got my index through API, it seems that severity is not defined as a mulit-field with analyzed. It is not_analyzed instead. Why it can not be sorted?

"severity":{"type":"string","norms":{"enabled":false},"fielddata":{"for
mat":"disabled"},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}}

[tsullivan (Tim Sullivan)]

Hi jenny8,

Sorry for the delay in response, but 2.1 is pretty old and looking around in this kind of environment is a huge context switch.

Based on the mapping you provided, your severity field looks like it is a multi-field, and if I reproduce a simple case of an index with only a field like that, it would be:

PUT /my_index
{
  "mappings": {
    "_default_": {
      "_timestamp": {
        "enabled": true
      }
    },
    "doc": {
      "properties": {
        "severity": {
          "type": "string",
          "norms": {
            "enabled": false
          },
          "fielddata": {
            "format": "disabled"
          },
          "fields": {
            "raw": {
              "type": "string",
              "index": "not_analyzed",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

###

POST /my_index/doc
{
	"severity": "abc"
}

POST /my_index/doc
{
	"severity": "bcd"
}

POST /my_index/doc
{
	"severity": "zyx"
}

POST /my_index/doc
{
	"severity": "srq"
}

POST /my_index/doc/_search
{
  "sort": {
    "severity.raw": "asc" 
  }
}

The sort does correctly return the documents in the right order, but that's in pure ES, not Kibana. I noticed that when I tried to make this into an Index Pattern in Kibana, the Discover app wasn't able to sort the documents based on severity. I think that is because the .raw field isn't reflected in the _source and Kibana doesn't seem to be aware of it. But that field should be available as a not_analyzed field in visualizations.

Just curious, but if there are barriers in the way of upgrading to a newer stack version, would you mind sharing what they are?