Upgraded to ELK 5.0 - can no longer create Visualizations with Aggregations


(Paul Ainslie) #1

I upgraded from ES 2.4 to 5.0 this week.

When I try to create Visualizations with Aggregations in Kibana, the only numeric fields that are available to me are the scripted fields and none of the string fields are available. Yet the other indexes seem to work fine in ES & Kibana 5.0!

I noticed that none of the fields are 'aggregatable'. Is that why I am unable to create Visualizations with Aggregations?

Here's the index definition (part of it, as I could not fit it all in this post).

{
  "mediaserver" : {
    "mappings" : {
      "usage" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "@version" : {
            "type" : "string"
          },
          "account" : {
            "type" : "string",
            "index" : "not_analyzed",
            "fielddata" : false
          },
          "agent" : {
            "type" : "string"
          },
          "bytes" : {
            "type" : "long"
          },
          "cache_status" : {
            "type" : "string",
            "index" : "not_analyzed",
            "fielddata" : false
          },
          "clientip" : {
            "type" : "ip"
          },
          "geoip" : {
            "properties" : {
              "area_code" : {
                "type" : "short"
              },
              "city_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "continent_code" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "country_code2" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "country_code3" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "country_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "dma_code" : {
                "type" : "short"
              },
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "double"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "double"
              },
              "postal_code" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "real_region_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "region_name" : {
                "type" : "string",
                "index" : "not_analyzed",
                "fielddata" : false
              },
              "timezone" : {
                "type" : "string"
              }
            }
          },

(Paul Ainslie) #2

I figured it out, when it converted to ES 5.0 it set searchable and aggregatable to false. Argh! Not sure what it did that.

I guess I'll have to create a new index and do a migration.


(Robert Hafner) #3

Had this happen to me as well. It's an amazingly annoying bug. Is there a way to fix it other than creating new indexes and migrating over? A way to enable it for specific fields, for instance?


(Paul Ainslie) #4

Sorry, you can't change the fields, you have to change them to something like,
"filename" : { "type" : "keyword", "index": "not_analyzed" },
This can only be done by creating a brand new index, and then initiate a reindex job: to migrate your data like this.

curl -XPOST localhost:9200/_reindex?pretty -d @my_reindex_file.json

Input file

{
  "source": {
    "index": "myindex_v1"
  },
  "dest": {
    "index": "myindex_v2"
  }
}

Then check status on running tasks:
curl -XGET 'localhost:9200/_tasks?detailed=true&actions=*reindex&pretty'

Actually, I've reindexed so many times now I have logstash and outside applications use an alias so they don't have to keep changing.


(system) #5