Hi,
We are using ES for half a year, and we are very pleased with how it's been
working. We have 3 servers running sharded ES configuration. We have two
mappings for two different document types. Recently, we dropped one of the
mappings and then later, added it back. We think that the problem we are
facing started since that single mapping was recreated (although we aren't
100% sure). We were hoping you can help us with this error we are
experiencing. When running a facet query, in the following form:
'{"facets":{"bounce_types":{"terms":{"field":"BT","size":100},"facet_filter":{"term":{"SID":50197}}}}}'
We get shard failures that contain two types of exceptions:
RemoteTransportException and QueryPhaseExecutionException. Here is how they
look:
{
"took" : 1120,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 3,
"failed" : 7,
"failures" : [ {
"index" : "myapp_sharded_prod",
"shard" : 1,
"status" : 500,
"reason" :
"RemoteTransportException[[Glitch][inet[/172.30.0.214:9300]][search/phase/query]];
nested: QueryPhaseExecutionException[[myapp_sharded_prod][1]:
query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]:
Query Failed [Failed to execute main query]]; nested:
NumberFormatException[Invalid shift value in prefixCoded string (is encoded
value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 0,
"status" : 500,
"reason" :
"RemoteTransportException[[Glitch][inet[/172.30.0.214:9300]][search/phase/query]];
nested: QueryPhaseExecutionException[[myapp_sharded_prod][0]:
query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]:
Query Failed [Failed to execute main query]]; nested:
NumberFormatException[Invalid shift value in prefixCoded string (is encoded
value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 3,
"status" : 500,
"reason" : "RemoteTransportException[[Justin
Hammer][inet[/172.30.0.133:9300]][search/phase/query]]; nested:
QueryPhaseExecutionException[[myapp_sharded_prod][3]:
query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]:
Query Failed [Failed to execute main query]]; nested:
NumberFormatException[Invalid shift value in prefixCoded string (is encoded
value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 9,
"status" : 500,
"reason" : "QueryPhaseExecutionException[[myapp_sharded_prod][9]:
query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]:
Query Failed [Failed to execute main query]]; nested:
NumberFormatException[Invalid shift value in prefixCoded string (is encoded
value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 8,
"status" : 500,
"reason" : "QueryPhaseExecutionException[[myapp_sharded_prod][8]:
query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]:
Query Failed [Failed to execute main query]]; nested:
NumberFormatException[Invalid shift value in prefixCoded string (is encoded
value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 4,
"status" : 500,
"reason" : "RemoteTransportException[[Justin
Hammer][inet[/172.30.0.133:9300]][search/phase/query]]; nested:
QueryPhaseExecutionException[[myapp_sharded_prod][4]:
query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]:
Query Failed [Failed to execute main query]]; nested:
NumberFormatException[Invalid shift value in prefixCoded string (is encoded
value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 7,
"status" : 500,
"reason" : "QueryPhaseExecutionException[[myapp_sharded_prod][7]:
query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]:
Query Failed [Failed to execute main query]]; nested:
NumberFormatException[Invalid shift value in prefixCoded string (is encoded
value really an INT?)]; "
} ]
},
"hits" : {
"total" : 37387748,
"max_score" : 1.0,
"hits" : [ ]
},
"facets" : {
"bounce_types" : {
"_type" : "terms",
"missing" : 2600,
"total" : 95,
"other" : 0,
"terms" : [ {
"term" : 1,
"count" : 92
}, {
"term" : 64,
"count" : 2
}, {
"term" : 256,
"count" : 1
} ]
}
}
Can you provide us with any details on why these might be happening? Have
we put bad mapping back into ES, or is this something server-related (ES
version, JVM version)? I don't know if it is relevant, but we have two
mappings there and one was intact and the other one was recreated. All the
fields in mappings are present under the same names in both of them. If any
other info could help, please let me know and I'll post details right away.
Thanks a lot for your time!
Milan Gornik
--