# Embedded elasticsearch - ClassCastException when clustered **URL:** \ **Category:** Elasticsearch\ **Created:** [May 13, 2013, 8:59pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942 "2013-05-13T20:59:59Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![esadhir](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@esadhir](https://discuss.elastic.co/u/esadhir)\ **Post date:** [May 13, 2013, 8:59pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/1 "2013-05-13T20:59:59Z")
We have a java application which embeds elasticsearch instances on multiple servers, and starts them as follows: node = nodeBuilder().client(false).local(false).node(); client = node.client(); //wait for ready client.admin().cluster().health(newClusterHealthRequest().waitForYellowStatus()).actionGet(); This work fine when started on a single box, and the application is able to put/get records without issue, and passes benchmarks/stress tests. When we start the application on multiple nodes, with elasticsearch.yml configured to allow the nodes to find each other via unicast, the cluster comes up fine and goes green immediately. However, get requests randomly start failing fairly quickly with a ClassCastException. Here is the relevant part of the exception: 2013-05-13 16:37:35,597 ERROR New I/O worker #15 Hazelbridge - Exception in ServerHandler java.lang.ClassCastException: org.elasticsearch.common.bytes.ChannelBufferBytesReference cannot be cast to org.elasticsearch.common.bytes.BytesArray This exception occurs at the following call: GetResponse g = client.prepareGet(esIndex, namespace, newString(key)).setFields( "expiry","data").execute().actionGet(); if (g.isExists()) { int exp = (int) g.getField("expiry").getValue(); BytesArray b = (BytesArray) g.getField("data").getValue(); //THIS LINE CAUSES THE EXCEPTION } The value of exp is retrieved successfully and it can be logged to verify. This doesn't happen for the first 3 or 4 puts, but then it starts happening on 75% of gets. This does not happen in the single node case. Help? The mapping for the type is set up with the following code: * * * String mapping = jsonBuilder() .startObject() .startObject(ns) .startObject("\_ttl") .field("enabled","true") .endObject() .startObject("\_source") .field("enabled","false") .endObject() .startObject("\_all") .field("enabled", "false") .endObject() .startObject("properties") .startObject("data") .field("type","binary") .field("index","not\_analyzed") .field("store","yes") .field("compress","false") .endObject() .startObject("expiry") .field("type","integer") .field("index","not\_analyzed") .field("store","yes") .field("compress","false") .endObject() .endObject() .endObject() .string(); client.admin().indices().preparePutMapping(esIndex ).setType(ns).setSource(mapping).execute().actionGet(); -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![esadhir](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@esadhir](https://discuss.elastic.co/u/esadhir)\ **Post date:** [May 16, 2013, 5:12pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/2 "2013-05-16T17:12:39Z")
Noone? On Monday, May 13, 2013 4:59:59 PM UTC-4, esadhir wrote: > We have a java application which embeds elasticsearch instances on > multiple servers, and starts them as follows: > > node = nodeBuilder().client(false).local(false).node(); > > client = node.client(); > > //wait for ready > > client.admin().cluster().health(newClusterHealthRequest().waitForYellowStatus()).actionGet(); > > This work fine when started on a single box, and the application is able > to put/get records without issue, and passes benchmarks/stress tests. > > When we start the application on multiple nodes, with elasticsearch.yml > configured to allow the nodes to find each other via unicast, the cluster > comes up fine and goes green immediately. However, get requests randomly > start failing fairly quickly with a ClassCastException. Here is the > relevant part of the exception: > > 2013-05-13 16:37:35,597 ERROR New I/O worker #15 Hazelbridge - Exception > in ServerHandler java.lang.ClassCastException: > org.elasticsearch.common.bytes.ChannelBufferBytesReference cannot be cast > to org.elasticsearch.common.bytes.BytesArray > > This exception occurs at the following call: > > GetResponse g = client.prepareGet(esIndex, namespace, newString(key)).setFields( > "expiry","data").execute().actionGet(); > > if (g.isExists()) { > > int exp = (int) g.getField("expiry").getValue(); > BytesArray b = (BytesArray) g.getField("data").getValue(); //THIS LINE > CAUSES THE EXCEPTION > > } > > The value of exp is retrieved successfully and it can be logged to verify. > > This doesn't happen for the first 3 or 4 puts, but then it starts > happening on 75% of gets. This does not happen in the single node case. > > Help? > > The mapping for the type is set up with the following code: > > * * * > > String mapping = jsonBuilder() > > .startObject() > > .startObject(ns) > > .startObject("\_ttl") > > .field("enabled","true") > > .endObject() > > .startObject("\_source") > > .field("enabled","false") > > .endObject() > > .startObject("\_all") > > .field("enabled", "false") > > .endObject() > > .startObject("properties") > > .startObject("data") > > .field("type","binary") > > .field("index","not\_analyzed") > > .field("store","yes") > > .field("compress","false") > > .endObject() > > .startObject("expiry") > > .field("type","integer") > > .field("index","not\_analyzed") > > .field("store","yes") > > .field("compress","false") > > .endObject() > > .endObject() > > .endObject() > > .string(); > > client.admin().indices().preparePutMapping(esIndex > ).setType(ns).setSource(mapping).execute().actionGet(); -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [May 16, 2013, 5:21pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/3 "2013-05-16T17:21:39Z")
Hey, sorry for not getting back. I will try to reproduce your issue tomorrow. Do you have only entities indexed, which have both fields set? Also can you provide the full exception from your logfile? Can you tell us your elasticsearch version as well? --Alex On Thu, May 16, 2013 at 7:12 PM, esadhir [alokdhir@gmail.com](mailto:alokdhir@gmail.com) wrote: > Noone? > > On Monday, May 13, 2013 4:59:59 PM UTC-4, esadhir wrote: > > > We have a java application which embeds elasticsearch instances on > > multiple servers, and starts them as follows: > > > > node = nodeBuilder().client(false).\*\*local(false).node(); > > > > client = node.client(); > > > > //wait for ready > > > > client.admin().cluster().**health(new ClusterHealthRequest().** > > waitForYellowStatus()).\*\*actionGet(); > > > > This work fine when started on a single box, and the application is able > > to put/get records without issue, and passes benchmarks/stress tests. > > > > When we start the application on multiple nodes, with elasticsearch.yml > > configured to allow the nodes to find each other via unicast, the cluster > > comes up fine and goes green immediately. However, get requests randomly > > start failing fairly quickly with a ClassCastException. Here is the > > relevant part of the exception: > > > > 2013-05-13 16:37:35,597 ERROR New I/O worker #15 Hazelbridge - Exception > > in ServerHandler java.lang.ClassCastException: org.elasticsearch.common.\* > > \*bytes.\*\*ChannelBufferBytesReference cannot be cast to > > org.elasticsearch.common.\*\*bytes.BytesArray > > > > This exception occurs at the following call: > > > > GetResponse g = client.prepareGet(esIndex, namespace, newString(key)).setFields( > > "\*\*expiry","data").execute().\*\*actionGet(); > > > > if (g.isExists()) { > > > > int exp = (int) g.getField("expiry").getValue(\*\*); > > BytesArray b = (BytesArray) g.getField("data").getValue(); //THIS LINE > > CAUSES THE EXCEPTION > > > > } > > > > The value of exp is retrieved successfully and it can be logged to verify. > > > > This doesn't happen for the first 3 or 4 puts, but then it starts > > happening on 75% of gets. This does not happen in the single node case. > > > > Help? > > > > The mapping for the type is set up with the following code: > > > > * * * > > > > String mapping = jsonBuilder() > > > > .startObject() > > > > .startObject(ns) > > > > .startObject("\_ttl") > > > > .field("enabled","true") > > > > .endObject() > > > > .startObject("\_source") > > > > .field("enabled","false") > > > > .endObject() > > > > .startObject("\_all") > > > > .field("enabled", "false") > > > > .endObject() > > > > .startObject("properties") > > > > .startObject("data") > > > > .field("type","binary") > > > > .field("index","not\_analyzed") > > > > .field("store","yes") > > > > .field("compress","false") > > > > .endObject() > > > > .startObject("expiry") > > > > .field("type","integer") > > > > .field("index","not\_analyzed") > > > > .field("store","yes") > > > > .field("compress","false") > > > > .endObject() > > > > .endObject() > > > > .endObject() > > > > .string(); > > > > client.admin().indices().**preparePutMapping(esIndex).** > > setType(ns).setSource(mapping)\*\*.execute().actionGet(); > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [May 17, 2013, 2:39pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/4 "2013-05-17T14:39:46Z")
Hey, first I thought, you found a bug in elasticsearch, but it is actually a bug in your code. You are casting to the wrong class. You need to cast to BytesReference instead of BytesArray. Short explanation long: BytesReference is the interface used to abstract. BytesArray is used, if the GetRequest got the data from a shard from the node the query was issued at (because there is no need to serialize and sent data over the wire). ChannelBufferBytesReference is used, in case the data was fetched from a shard from another node in your cluster. This also explains why you are seeing this exception not all the time, but only 75% of the time (I guess you have a four node setup then). Hope this helps and have a nice weekend! --Alex On Thu, May 16, 2013 at 7:21 PM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > Hey, > > sorry for not getting back. I will try to reproduce your issue tomorrow. > Do you have only entities indexed, which have both fields set? > Also can you provide the full exception from your logfile? Can you tell us > your elasticsearch version as well? > > --Alex > > On Thu, May 16, 2013 at 7:12 PM, esadhir [alokdhir@gmail.com](mailto:alokdhir@gmail.com) wrote: > > > Noone? > > > > On Monday, May 13, 2013 4:59:59 PM UTC-4, esadhir wrote: > > > > > We have a java application which embeds elasticsearch instances on > > > multiple servers, and starts them as follows: > > > > > > node = nodeBuilder().client(false).\*\*local(false).node(); > > > > > > client = node.client(); > > > > > > //wait for ready > > > > > > client.admin().cluster().**health(new ClusterHealthRequest().** > > > waitForYellowStatus()).\*\*actionGet(); > > > > > > This work fine when started on a single box, and the application is able > > > to put/get records without issue, and passes benchmarks/stress tests. > > > > > > When we start the application on multiple nodes, with elasticsearch.yml > > > configured to allow the nodes to find each other via unicast, the cluster > > > comes up fine and goes green immediately. However, get requests randomly > > > start failing fairly quickly with a ClassCastException. Here is the > > > relevant part of the exception: > > > > > > 2013-05-13 16:37:35,597 ERROR New I/O worker #15 Hazelbridge - Exception > > > in ServerHandler java.lang.ClassCastException: org.elasticsearch.common. > > > \*\*bytes.\*\*ChannelBufferBytesReference cannot be cast to > > > org.elasticsearch.common.\*\*bytes.BytesArray > > > > > > This exception occurs at the following call: > > > > > > GetResponse g = client.prepareGet(esIndex, namespace, newString(key)).setFields( > > > "\*\*expiry","data").execute().\*\*actionGet(); > > > > > > if (g.isExists()) { > > > > > > int exp = (int) g.getField("expiry").getValue(\*\*); > > > BytesArray b = (BytesArray) g.getField("data").getValue(); //THIS LINE > > > CAUSES THE EXCEPTION > > > > > > } > > > > > > The value of exp is retrieved successfully and it can be logged to > > > verify. > > > > > > This doesn't happen for the first 3 or 4 puts, but then it starts > > > happening on 75% of gets. This does not happen in the single node case. > > > > > > Help? > > > > > > The mapping for the type is set up with the following code: > > > > > > * * * > > > > > > String mapping = jsonBuilder() > > > > > > .startObject() > > > > > > .startObject(ns) > > > > > > .startObject("\_ttl") > > > > > > .field("enabled","true") > > > > > > .endObject() > > > > > > .startObject("\_source") > > > > > > .field("enabled","false") > > > > > > .endObject() > > > > > > .startObject("\_all") > > > > > > .field("enabled", "false") > > > > > > .endObject() > > > > > > .startObject("properties") > > > > > > .startObject("data") > > > > > > .field("type","binary") > > > > > > .field("index","not\_analyzed") > > > > > > .field("store","yes") > > > > > > .field("compress","false") > > > > > > .endObject() > > > > > > .startObject("expiry") > > > > > > .field("type","integer") > > > > > > .field("index","not\_analyzed") > > > > > > .field("store","yes") > > > > > > .field("compress","false") > > > > > > .endObject() > > > > > > .endObject() > > > > > > .endObject() > > > > > > .string(); > > > > > > client.admin().indices().**preparePutMapping(esIndex).** > > > setType(ns).setSource(mapping)\*\*.execute().actionGet(); > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![esadhir](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@esadhir](https://discuss.elastic.co/u/esadhir)\ **Post date:** [May 31, 2013, 5:10pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/5 "2013-05-31T17:10:26Z")
Thank you for your response - I'll test and get back. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![esadhir](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@esadhir](https://discuss.elastic.co/u/esadhir)\ **Post date:** [June 3, 2013, 7:30pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/6 "2013-06-03T19:30:49Z")
The fix worked great - thank you! On Friday, May 31, 2013 1:10:26 PM UTC-4, esadhir wrote: > Thank you for your response - I'll test and get back. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![esadhir](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@esadhir](https://discuss.elastic.co/u/esadhir)\ **Post date:** [August 15, 2013, 8:26pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/7 "2013-08-15T20:26:36Z")
New issue with this - same environment. After several days of working fine, clustered, using the (BytesReference) cast you suggested, it starts failing again - but alternating (see below). Inexplicably, after some period of time - usually days, the values we're storing, which are binary, start failing with: Aug 15 15:56:04 localhost.localdomain ERROR: Hazelbridge - Exception in ServerHandler java.lang.ClassCastException: java.lang.String cannot be cast to org.elasticsearch.common.bytes.BytesReference Aug 15 15:56:04 localhost.localdomain ERROR: Hazelbridge - ..com.symplicity.hazelbridge.backends.elasticsearch.ElasticsearchInstance.get(ElasticsearchInstance.java:163) That occurs at this line: BytesReference b = (BytesReference) g.getField("data").getValue(); The value at this point has somehow been changed into a string, even though it was originally put in as binary, and verified via the mapping api. This is further verified by pulling this data via the HTTP API. Usually when pulling binary records, the binary data is not returned and you simply see "exists: true" in the return json. After the "corruption" has occurred, it returns the base 64 encoded binary string. Internally somehow the stored data has changed types. Interestingly, when this happens, the "expiry" field in the same record has also spontaneously changed from an int to a long, even though in our code we never treat that value as a long. These two conditions always occur on the same record - i.e. once the bug shows up, both fields will have switched to the wrong data type. This is an example of that occurring: Aug 15 15:56:04 localhost.localdomain ERROR: Hazelbridge - Exception in ServerHandler java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer Aug 15 15:56:04 localhost.localdomain ERROR: Hazelbridge - ..com.symplicity.hazelbridge.backends.elasticsearch.ElasticsearchInstance.get(ElasticsearchInstance.java:152) Also, sometimes, it actually cycles between the wrong type and the right in subsequent get requests - even to the same cluster node. If I had to guess, something is amiss in shard migration perhaps - the record's associated mapping is somehow lost on one of the nodes. Guessing this since it only happens after a while of running. Any help would be appreciated. Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![esadhir](https://avatars.discourse-cdn.com/v4/letter/e/278dde/32.png) [@esadhir](https://discuss.elastic.co/u/esadhir)\ **Post date:** [September 23, 2013, 4:29pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/8 "2013-09-23T16:29:33Z")
> bump -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Kirk\_Tsai](https://avatars.discourse-cdn.com/v4/letter/k/2bfe46/32.png) [@Kirk\_Tsai](https://discuss.elastic.co/u/Kirk_Tsai)\ **Post date:** [April 27, 2016, 8:19pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/9 "2016-04-27T20:19:05Z")
I am seeing exact same behavior as you described. Wondering if you ever figured out the problem. Caused by: ! java.lang.ClassCastException: java.lang.String cannot be cast to org.elasticsearch.common.bytes.BytesReference The offending line of code is: BytesArray bytesArray = ((BytesReference)resp .getField( IndexSchema.BLOOM\_FILTER) .getValue()) .toBytesArray(); The BLOOK\_FILTER field is defined as binary type. Not sure why query returned String type. I am running ElasticSearch Version 1.3.4. Thanks, Kirk ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:55pm UTC](https://discuss.elastic.co/t/embedded-elasticsearch-classcastexception-when-clustered/11942/10 "2017-07-05T22:55:45Z")