NumberFormatException doing a query we've done tons of times (from kibana)

Elastic Stack Elasticsearch
1

I have an index that's been working fine for over 2 years. This evening, a query we do all the time, from a Kibana3 dashboard fails 100% of the time. Similar dashboard panel in Kibana4 is failing.

The cluster is: 2b12e0

And it's this index: activeusers-production

In Kibana we are summing across the active summary documents. Query fails when run in kibana of course, and you can cut/paste it into the Any Request tab targeting POST /activeusers-production/activesummary/_search

Error is:
"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[7i-mRfIfTk-KzwF-AgTsQg][activeusers-production][0]: RemoteTransportException[[instance-0000000071][inet[/172.17.0.13:19320]][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[[activeusers-production][0]: query[ConstantScore(cache(_type:activesummary))],from[0],size[0]: Query Failed [Failed to execute global facets]]; nested: ElasticsearchException[java.lang.NumberFormatException: Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)];

{
"facets": {
"0": {
"date_histogram": {
"key_field": "@timestamp",
"value_field": "1d_unique_clients",
"interval": "1d"
},
"global": true,
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "_type:activesummary AND steller_product_base:Steller"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1459134116422,
"to": 1461726116423
}
}
}
]
}
}
}
}
}
}
},
"1": {
"date_histogram": {
"key_field": "@timestamp",
"value_field": "1d_unique_clients",
"interval": "1d"
},
"global": true,
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "_type:activesummary AND steller_product_base:Player"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1459134116424,
"to": 1461726116424
}
}
}
]
}
}
}
}
}
}
},
"3": {
"date_histogram": {
"key_field": "@timestamp",
"value_field": "1d_unique_clients",
"interval": "1d"
},
"global": true,
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "_type:activesummary AND steller_product_base:StellerAndroid"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1459134116424,
"to": 1461726116424
}
}
}
]
}
}
}
}
}
}
}
},
"size": 0
}

2

Does it make sense that this could be a partial cause?

All of the documents that I am working against are of type activesummary, see mapping below.

"activesummary": {
"dynamic_templates": [
{
"string_fields": {
"mapping": {
"index": "not_analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"properties": {
"28d_unique_users": {
"type": "integer"
},
"steller_product_base": {
"index": "not_analyzed",
"type": "string"
},
"@timestamp": {
"format": "dateOptionalTime",
"type": "date"
},
"7d_unique_clients": {
"type": "integer"
},
"28d_unique_clients": {
"type": "integer"
},
"@version": {
"index": "not_analyzed",
"type": "string"
},
"1d_unique_clients": {
"type": "integer"
},
"1d_unique_users": {
"type": "integer"
},
"7d_unique_users": {
"type": "integer"
}
}
},

A few days ago, due to an inadvertent dest run, a mapping for activesummaryXX was auto-created. It has a similar shape, but a completely different document and there should be no documents of that type in the index, BUT it does have some fields that map to long in the bogus document and integer in the real "activesummary" document.

Regenerating the entire index is not possible so I'd like to understand how we can recover if this is the issue.

"activesummaryXX": {
"dynamic_templates": [
{
"string_fields": {
"mapping": {
"index": "not_analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"properties": {
"28d_unique_users": {
"type": "long"
},
"steller_product_base": {
"index": "not_analyzed",
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
}
},
"@timestamp": {
"format": "dateOptionalTime",
"type": "date"
},
"7d_unique_clients": {
"type": "long"
},
"28d_unique_clients": {
"type": "long"
},
"@version": {
"index": "not_analyzed",
"type": "string"
},
"1d_unique_clients": {
"type": "long"
},
"1d_unique_users": {
"type": "long"
},
"7d_unique_users": {
"type": "long"
}
}
},