Upgraded to ELK 5.0 - can no longer create Visualizations with Aggregations

Elastic Stack Elasticsearch
1

I upgraded from ES 2.4 to 5.0 this week.

When I try to create Visualizations with Aggregations in Kibana, the only numeric fields that are available to me are the scripted fields and none of the string fields are available. Yet the other indexes seem to work fine in ES & Kibana 5.0!

Screen Shot 2016-11-02 at 6.01.31 PM.png820×1372 72 KB

I noticed that none of the fields are 'aggregatable'. Is that why I am unable to create Visualizations with Aggregations?

Here's the index definition (part of it, as I could not fit it all in this post).

{
  "mediaserver" : {
    "mappings" : {
      "usage" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "@version" : {
            "type" : "string"
          },
          "account" : {
            "type" : "string",
            "index" : "not_analyzed",
            "fielddata" : false
          },
          "agent" : {
            "type" : "string"
          },
          "bytes" : {
            "type" : "long"
          },
          "cache_status" : {
            "type" : "string",
            "index" : "not_analyzed",
            "fielddata" : false
          },
          "clientip" : {
            "type" : "ip"
          },
          "geoip" : {
            "properties" : {
              "area_code" : {
                "type" : "short"
              },
              "city_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "continent_code" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "country_code2" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "country_code3" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "country_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "dma_code" : {
                "type" : "short"
              },
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "double"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "double"
              },
              "postal_code" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "real_region_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "region_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "timezone" : {
                "type" : "string"
              }
            }
          },
2

I figured it out, when it converted to ES 5.0 it set searchable and aggregatable to false. Argh! Not sure what it did that.

I guess I'll have to create a new index and do a migration.

Screen Shot 2016-11-03 at 8.21.28 AM.png1620×1192 75.7 KB

3

Had this happen to me as well. It's an amazingly annoying bug. Is there a way to fix it other than creating new indexes and migrating over? A way to enable it for specific fields, for instance?

4

Sorry, you can't change the fields, you have to change them to something like,
"filename" : { "type" : "keyword", "index": "not_analyzed" },
This can only be done by creating a brand new index, and then initiate a reindex job: to migrate your data like this.

curl -XPOST localhost:9200/_reindex?pretty -d @my_reindex_file.json

Input file

{
  "source": {
    "index": "myindex_v1"
  },
  "dest": {
    "index": "myindex_v2"
  }
}

Then check status on running tasks:
curl -XGET 'localhost:9200/_tasks?detailed=true&actions=*reindex&pretty'

Actually, I've reindexed so many times now I have logstash and outside applications use an alias so they don't have to keep changing.