Hi,
We are using ES for half a year, and we are very pleased with how it's been working. We have 3 servers running sharded ES configuration. We have two mappings for two different document types. Recently, we dropped one of the mappings and then later, added it back. We think that the problem we are facing started since that single mapping was recreated (although we aren't 100% sure). We were hoping you can help us with this error we are experiencing. When running a facet query, in the following form:
'{"facets":{"bounce_types":{"terms":{"field":"BT","size":100},"facet_filter":{"term":{"SID":50197}}}}}'
We get shard failures that contain two types of exceptions: RemoteTransportException and QueryPhaseExecutionException. Here is how they look:
{
"took" : 1120,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 3,
"failed" : 7,
"failures" : [ {
"index" : "myapp_sharded_prod",
"shard" : 1,
"status" : 500,
"reason" : "RemoteTransportException[[Glitch][inet[/172.30.0.214:9300]][search/phase/query]]; nested: QueryPhaseExecutionException[[myapp_sharded_prod][1]: query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: NumberFormatException[Invalid shift value in prefixCoded string (is encoded value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 0,
"status" : 500,
"reason" : "RemoteTransportException[[Glitch][inet[/172.30.0.214:9300]][search/phase/query]]; nested: QueryPhaseExecutionException[[myapp_sharded_prod][0]: query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: NumberFormatException[Invalid shift value in prefixCoded string (is encoded value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 3,
"status" : 500,
"reason" : "RemoteTransportException[[Justin Hammer][inet[/172.30.0.133:9300]][search/phase/query]]; nested: QueryPhaseExecutionException[[myapp_sharded_prod][3]: query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: NumberFormatException[Invalid shift value in prefixCoded string (is encoded value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 9,
"status" : 500,
"reason" : "QueryPhaseExecutionException[[myapp_sharded_prod][9]: query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: NumberFormatException[Invalid shift value in prefixCoded string (is encoded value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 8,
"status" : 500,
"reason" : "QueryPhaseExecutionException[[myapp_sharded_prod][8]: query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: NumberFormatException[Invalid shift value in prefixCoded string (is encoded value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 4,
"status" : 500,
"reason" : "RemoteTransportException[[Justin Hammer][inet[/172.30.0.133:9300]][search/phase/query]]; nested: QueryPhaseExecutionException[[myapp_sharded_prod][4]: query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: NumberFormatException[Invalid shift value in prefixCoded string (is encoded value really an INT?)]; "
}, {
"index" : "myapp_sharded_prod",
"shard" : 7,
"status" : 500,
"reason" : "QueryPhaseExecutionException[[myapp_sharded_prod][7]: query[ConstantScore(NotDeleted(cache(_type:DeliveryEvent)))],from[0],size[0]: Query Failed [Failed to execute main query]]; nested: NumberFormatException[Invalid shift value in prefixCoded string (is encoded value really an INT?)]; "
} ]
},
"hits" : {
"total" : 37387748,
"max_score" : 1.0,
"hits" : [ ]
},
"facets" : {
"bounce_types" : {
"_type" : "terms",
"missing" : 2600,
"total" : 95,
"other" : 0,
"terms" : [ {
"term" : 1,
"count" : 92
}, {
"term" : 64,
"count" : 2
}, {
"term" : 256,
"count" : 1
} ]
}
}
Can you provide us with any details on why these might be happening? Have we put bad mapping back into ES, or is this something server-related (ES version, JVM version)? I don't know if it is relevant, but we have two mappings there and one was intact and the other one was recreated. All the fields in mappings are present under the same names in both of them. If any other info could help, please let me know and I'll post details right away.
Thanks a lot for your time!
Milan Gornik