Hello hello! I have a bizarre error I've been trying to debug for a few
weeks with no luck, and I'm finally left to conclude that it may be a bug
in ElasticSearch.
Once every few days, I start seeing shard failures in my query results,
like this:
{
"index": "my_index",
"shard": 3,
"status": 500,
"reason": "RemoteTransportException[[HOSTNAME][inet[/10.0.123.123:9300]][search/phase/query]]; nested: QueryPhaseExecutionException[[my_index][3]: query[ConstantScore(cache(_type:my_type))],from[0],size[10],sort[<custom:"id": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@97c2b4f>]: Query Failed [Failed to execute main query]]; nested: ElasticSearchException[java.lang.NumberFormatException: Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)]; nested: UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)]; nested: NumberFormatException[Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)]; "
}
This query is operating against an index with about 100 different fields
(including several different nested types), but the relevant portion of the
mapping looks like this:
{
"my_type" : {
"_id" : { "type" : "long", "path" : "id" },
"properties" : {
"id" : { "type" : "long" },
/* ... LOTS OF OTHER FIELDS, INCLUDING MANY NESTED TYPES */
}}
}
I've been able to isolate the shard failures to a minimal query of this
form:
{
"query" : { "match_all" : { } },
"sort" : [{
"id" : { "order" : "asc" }
}]
}
Basically, sorting by (numeric) ID causes shard failures when the shards
sometimes mistakenly think that there are non-numeric values in the "id"
field. I've audited the data, and it conforms with the schema. The id
fields always contain valid LONG values.
Whenever the shard failures occur, I can silence them for a few days by
optimizing the index, like this:
curl -XPOST 'http://HOSTNAME:9200/my_index/_optimize?max_num_segments=1'
And the shard failures will stop for a day or two, but inevitably, within a
few days the failures will return and I'll have to optimize the index
again. The weird thing is that the status URL always reports GREEN status
and all shards healthy, even when these queries are failing on every
request.
I experienced these failures originally on 0.90.5, but I continued seeing
the same problems after recently upgrading to 0.90.10. I even deleted the
index and rebuilt from scratch under 0.90.10, but I've kept seeing the same
failures.
Any idea what might be going on?
Thanks!
benji smith
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3d8da85f-ed52-4127-b283-94998e851713%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.