Kibana shows fields are not searchable and aggregatable

[CaptainFeng (Weidong Feng)]

Hi All

After upgrading from 2.x, all my fields(including raw) are not searchable and aggregatable.

Please help me to figure out why this happens?

[Brandon_Kobel]

@CaptainFeng which version of Kibana and Elasticsearch are you running now? Did you try refreshing the index pattern by click the "Refresh field list" button highlighted below in the red rectangle?

[CaptainFeng (Weidong Feng)]

Hi Brandon

The versions of Kibana and Elasticsearch are same as 5.4.0.

I tried to refresh or delete index pattern and rebuild it. But it didn't work for me.

And though these field look unsearchable and unaggregatable, I still can query them by Dev Tool in Kibana.

[Brandon_Kobel]

@CaptainFeng when you refresh your index pattern, do you see the following warning at the top of your screen?

[CaptainFeng (Weidong Feng)]

@Brandon_Kobel
No. There is no warning.

And it seems that mappings of old indices changed.

          "8103": {
        "type": "string",
        "norms": false,
        "fields": {
          "raw": {
            "type": "string",
            "index": "not_analyzed",
            "ignore_above": 256,
            "fielddata": false
          }
        }
      },

Did this will make 8103.raw field not searchable and unaggregatable?

[whoami]

by the way, what kind of field type is searchable or aggregatable? or is 'searchable and aggregatable' related to field type?

[Brandon_Kobel]

@CaptainFeng we use the mappings in Elasticsearch to figure out the types of the fields, and whether they're analyzed. However, we use the Field Capabilities API to determine whether fields are searchable/aggregatable.

Would you mind executing a CURL request similar to the following against Elasticsearch and posting the raw response here? curl http://localhost:9200/logstash-*/_field_caps?fields=* -u elastic:changeme. You'll want to replace logstash-* with your index pattern and the elasticsearch URL and username/password will likely need to change.

[Brandon_Kobel]

@whoami searchable/aggregatable are directly tied to the field capabilities as specified by Elasticsearch

[CaptainFeng (Weidong Feng)]

@Brandon_Kobel I executed field capabilities api.

    "8103": {
      "string": {
        "type": "string",
        "searchable": true,
        "aggregatable": true,
        "indices": [
            ...
          "logstash-2017.05.28",
          "logstash-2017.05.29",
          "logstash-2017.05.30",
          "logstash-2017.05.31",
          "logstash-2017.06.01"
        ]
      },
      "text": {
        "type": "text",
        "searchable": true,
        "aggregatable": false,
        "indices": [
          "logstash-2017.06.02",
          "logstash-2017.06.03",
          "logstash-2017.06.04",
          "logstash-2017.06.06",
          "logstash-2017.06.07"
        ]
      }
    }


    "8103.raw": {
  "string": {
    "type": "string",
    "searchable": true,
    "aggregatable": true
  }
}


    "8103.keyword": {
  "keyword": {
    "type": "keyword",
    "searchable": true,
    "aggregatable": true
  }
},

There are four different parts. It seems strange. I upgrade ES in 2017.06.01.

Thanks!

[whoami]

yes, but how the "field capabilities api " know which field is searchable or aggregatable? or can we user control it by mapping?

[Brandon_Kobel]

@whoami they're derived from the mappings.

[Brandon_Kobel]

@CaptainFeng Sorry, I'm not following the "four different parts", are you referring to us having 8103, 8103.raw and 8103.keyword?

[whoami]

thx, would you mind to give a example?