Terms aggregation fails on type "ip" fields

Elastic Stack Kibana
1

Hello,

I just converted some fields, which contained IP addresses, from type "string" to type "ip", using a template.
Now I want to create a split rows visualization including a terms aggregation on the dst_ip field: e.g. I want to see the Top 5 destination IP. While this was working until the dst_field was a string, it broke after I changed its type to "ip".
--> Kibana shows no results with the dst_ip field selected as the terms aggregation field, and an error if I select the dst_ip.raw:

Visualize: ClassCastException[org.elasticsearch.search.aggregations.bucket.terms.LongTerms$Bucket cannot be cast to org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket]

Since usually you can cast almost everything to a string, I am surprised to see a cast exception. Is this a bug or a desired behaviour? Should I reach my goal in a different way? I'd like to keep the "ip" type because it let me perform range queries, but I want to be able to run a terms aggregation on type "ip" fields too. Any suggestion?

Thanks,
Daniele

2

What version of Elasticsearch and Kibana are you using? Also, could you post sample records and mappings for your index? In my quick test on ES 1.6 and Kibana 4.1, a visualization with a Terms aggregation on ip type fields worked without a problem.

3

I'm having this same problem after updating a web server accesslog template to use "ip" as the type for a couple fields; visualizations using those fields now all say "No results displayed because all values equal 0."

ES 1.6
Kibana 4.1

Sample data:

{
  "_index": "logstash-apacheaccess-2015.07.21",
  "_type": "apacheaccess",
  "_id": "AU6xSiThWykxAv5usKyv",
  "_score": null,
  "_source": {
    "message": "<174>Jul 21 15:45:03 web15.mysite.com web15: 172.8.195.44 10.1.1.15 - - [21/Jul/2015:15:45:03 +0000] \"GET /assets/font/League_Gothic-    webfont.woff HTTP/1.1\" 200 22852 \"https://www.mysite.com/assets/compiled/css/asdf.css?1554726\" \"Mozilla/5.0 (Windows NT 6.3; WOW64)     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\" 0 www.mysite.com",
    "@version": "1",
    "@timestamp": "2015-07-21T15:45:03.000Z",
    "host": "10.1.1.10",
    "type": "apacheaccess",
    "webserver": "web15",
    "clientip": "172.8.195.44",
    "serverip": "10.1.1.15",
    "ident": "-",
    "auth": "-",
    "timestamp": "21/Jul/2015:15:45:03 +0000",
    "verb": "GET",
    "request": "/assets/font/League_Gothic-webfont.woff",
    "httpversion": "1.1",
    "response": "200",
    "bytes": 22852,
    "referrer": "\"https://www.mysite.com/assets/compiled/css/asdf.css?1554726\"",
    "agent": "\"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
    "timetaken": 0,
    "vhost": "www.mysite.com",
    "url_path": "/assets/font/League_Gothic-webfont.woff",
    "received_at": "2015-07-21T15:45:03.049Z",
    "received_from": "10.1.1.10",
    "ua_name": "Chrome",
    "ua_os": "Windows 8.1",
    "ua_os_name": "Windows 8.1",
    "ua_device": "Other",
    "ua_major": "43",
    "ua_minor": "0",
    "ua_patch": "2357"
  },
  "fields": {
    "received_at": [
      1437493503049
    ],
    "@timestamp": [
      1437493503000
    ]
  },
  "sort": [
    1437493503000
  ]
}

Mapping:

{
   "logstash-apacheaccess-2015.07.21": {
      "mappings": {
         "_default_": {
            "dynamic_templates": [
               {
                  "message_field": {
                     "mapping": {
                        "index": "analyzed",
                        "omit_norms": true,
                        "type": "string"
                     },
                     "match": "message",
                     "match_mapping_type": "string"
                  }
               },
               {
                  "string_fields": {
                     "mapping": {
                        "index": "analyzed",
                        "omit_norms": true,
                        "type": "string",
                        "fields": {
                           "raw": {
                              "ignore_above": 256,
                              "index": "not_analyzed",
                              "type": "string"
                           }
                        }
                     },
                     "match": "*",
                     "match_mapping_type": "string"
                  }
               }
            ],
            "_all": {
               "enabled": true
            },
            "properties": {
               "@version": {
                  "type": "string",
                  "index": "not_analyzed"
               },
               "geoip": {
                  "dynamic": "true",
                  "properties": {
                     "location": {
                        "type": "geo_point"
                     }
                  }
               }
            }
         },
         "apacheaccess": {
            "dynamic_templates": [
               {
                  "message_field": {
                     "mapping": {
                        "index": "analyzed",
                        "omit_norms": true,
                        "type": "string"
                     },
                     "match": "message",
                     "match_mapping_type": "string"
                  }
               },
               {
                  "string_fields": {
                     "mapping": {
                        "index": "analyzed",
                        "omit_norms": true,
                        "type": "string",
                        "fields": {
                           "raw": {
                              "ignore_above": 256,
                              "index": "not_analyzed",
                              "type": "string"
                           }
                        }
                     },
                     "match": "*",
                     "match_mapping_type": "string"
                  }
               }
            ],
            "_all": {
               "enabled": true
            },
            "properties": {
               "@timestamp": {
                  "type": "date",
                  "format": "dateOptionalTime"
               },
               "@type": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "@version": {
                  "type": "string",
                  "index": "not_analyzed"
               },
               "agent": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "auth": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "bytes": {
                  "type": "long"
               },
               "clientip": {
                  "type": "ip"
               },
               "geoip": {
                  "dynamic": "true",
                  "properties": {
                     "location": {
                        "type": "geo_point"
                     }
                  }
               },
               "host": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "httpversion": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "ident": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "message": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  }
               },
               "rawrequest": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "received_at": {
                  "type": "date",
                  "format": "dateOptionalTime"
               },
               "received_from": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "referrer": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "request": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "response": {
                  "type": "long"
               },
               "serverip": {
                  "type": "ip"
               },
               "timestamp": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "timetaken": {
                  "type": "long"
               },
               "type": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "ua_build": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "ua_device": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "ua_major": {
                  "type": "long"
               },
               "ua_minor": {
                  "type": "long"
               },
               "ua_name": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "ua_os": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "ua_os_major": {
                  "type": "long"
               },
               "ua_os_minor": {
                  "type": "long"
               },
               "ua_os_name": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "ua_patch": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "url_path": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "url_querystring": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "verb": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "vhost": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "webserver": {
                  "type": "string",
                  "norms": {
                     "enabled": false
                  },
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               }
            }
         }
      }
   }
}
4

I figured out my problem.

My visualizations were using the "x.raw" versions of the IP fields, which were no longer being created since the dynamic_mappings that creates them was only matching on string types.

That being said, I was able to create a visualization aggregating a newly typed field (string -> long) without incident. I did however have to reindex my data and during that process found that I needed to use the following settings on the indexes to prevent bad data from getting into the new indexes:

  "index.mapping.ignore_malformed": true,
  "index.mapping.coerce": true
5

Hi Tanya,

I somehow figured out my problem: I was searching on last week indices, and after my mapping change the field on which I was performing a terms aggregation was ambiguous. Some indices had it mapped as string while others as ip. This led to the cast exception. Now it is working as expected.

Thank you,
Daniele