Shard error on elasticsearch upgrade

Elastic Stack Elasticsearch
1

Finally upgrade our production cluster from 0.20.0RC2 to 0.90.2. Using the
old index built with 0.20, we started getting NumberFormatExceptions.

Example query:

"_shards": {
"total": 8,
"successful": 7,
"failed": 1,
"failures": [
{
"index": "foo",
"shard": 7,
"status": 500,
"reason":
"RemoteTransportException[[...][inet[/....159:9300]][search/phase/query]];
nested: QueryPhaseExecutionException[[foo][7]: query[...],from[0],size[40]:
Query Failed [Failed to execute main query]]; nested:
ElasticSearchException[java.lang.NumberFormatException: Invalid shift value
(64) in prefixCoded bytes (is encoded value really an INT?)]; nested:
UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift
value (64) in prefixCoded bytes (is encoded value really an INT?)]; nested:
NumberFormatException[Invalid shift value (64) in prefixCoded bytes (is
encoded value really an INT?)]; "
}
]
}

In this example, 1 shard out of 8 is failing because of an invalid shard.
We saw this issue when we upgraded our development cluster, but ignored it
because it was only the development cluster. A full reindex with the latest
version made the issue go away, and we are hoping the same holds true this
time around as well.

Cheers,

Ivan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I pruned the query until I was able to get a minimal query that would
break. The issue occurs while faceting on one specific field.

Failure:
{
"query" : {
"match_all" : { }
},
"facets" : {
"CatalogID" : {
"terms" : {
"field" : "CatalogID",
"size" : 20
}
}
}
}

Where the mapping is CatalogID: { type: long }

Querying on that field with no facets works fine. This facet is one of
many, most of which are long fields.

Cheers,

Ivan

On Fri, Jul 12, 2013 at 11:05 AM, Ivan Brusic ivan@brusic.com wrote:

Finally upgrade our production cluster from 0.20.0RC2 to 0.90.2. Using
the old index built with 0.20, we started getting NumberFormatExceptions.

Example query:

"_shards": {
"total": 8,
"successful": 7,
"failed": 1,
"failures": [
{
"index": "foo",
"shard": 7,
"status": 500,
"reason":
"RemoteTransportException[[...][inet[/....159:9300]][search/phase/query]];
nested: QueryPhaseExecutionException[[foo][7]: query[...],from[0],size[40]:
Query Failed [Failed to execute main query]]; nested:
ElasticSearchException[java.lang.NumberFormatException: Invalid shift value
(64) in prefixCoded bytes (is encoded value really an INT?)]; nested:
UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift
value (64) in prefixCoded bytes (is encoded value really an INT?)]; nested:
NumberFormatException[Invalid shift value (64) in prefixCoded bytes (is
encoded value really an INT?)]; "
}
]
}

In this example, 1 shard out of 8 is failing because of an invalid shard.
We saw this issue when we upgraded our development cluster, but ignored it
because it was only the development cluster. A full reindex with the latest
version made the issue go away, and we are hoping the same holds true this
time around as well.

Cheers,

Ivan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Ivan, thanks for sharing this find!
-Vinh

On Jul 12, 2013, at 11:23 AM, Ivan Brusic ivan@brusic.com wrote:

I pruned the query until I was able to get a minimal query that would break. The issue occurs while faceting on one specific field.

Failure:
{
"query" : {
"match_all" : { }
},
"facets" : {
"CatalogID" : {
"terms" : {
"field" : "CatalogID",
"size" : 20
}
}
}
}

Where the mapping is CatalogID: { type: long }

Querying on that field with no facets works fine. This facet is one of many, most of which are long fields.

Cheers,

Ivan

On Fri, Jul 12, 2013 at 11:05 AM, Ivan Brusic ivan@brusic.com wrote:
Finally upgrade our production cluster from 0.20.0RC2 to 0.90.2. Using the old index built with 0.20, we started getting NumberFormatExceptions.

Example query:

"_shards": {
"total": 8,
"successful": 7,
"failed": 1,
"failures": [
{
"index": "foo",
"shard": 7,
"status": 500,
"reason": "RemoteTransportException[[...][inet[/....159:9300]][search/phase/query]]; nested: QueryPhaseExecutionException[[foo][7]: query[...],from[0],size[40]: Query Failed [Failed to execute main query]]; nested: ElasticSearchException[java.lang.NumberFormatException: Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)]; nested: UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)]; nested: NumberFormatException[Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)]; "
}
]
}

In this example, 1 shard out of 8 is failing because of an invalid shard. We saw this issue when we upgraded our development cluster, but ignored it because it was only the development cluster. A full reindex with the latest version made the issue go away, and we are hoping the same holds true this time around as well.

Cheers,

Ivan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.