I have a setup with 2 servers:
openjdk version "1.7.0_06"
OpenJDK Runtime Environment (build 1.7.0_06-b24)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
elasticsearch-0.19.9
I have the following config on both:
cluster.name: elasticsearch
node.name: "death" (and on the other node.name "life")
discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.unicast.hosts: ["10.24.233.196","10.17.57.4"]
So I start adding documents to the 'life' node, and everything seems to be
going good except from time to time I get the below logs/exceptions on both
machines. Please tell me what I could be doing wrong.
[2012-09-26 08:45:15,963][WARN ][cluster.action.shard ] [death]
received shard failed for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw],
[R], s[STARTED], reason [
Failed to perform [index] on replica, message
[RemoteTransportException[Failed to deserialize exception response from
stream]; nested: TransportSerializationException[F
ailed to deserialize exception response from stream]; nested:
StreamCorruptedException[unexpected end of block data]; ]]
[2012-09-26 08:45:15,942][WARN ][action.index ] [life] Failed
to perform index on replica [prod_threads][0]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize exception response from stream
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:171)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:390)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.StreamCorruptedException: unexpected end of block data
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at
java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
at java.lang.Throwable.readObject(Throwable.java:913)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at
java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:498)
at java.lang.Throwable.readObject(Throwable.java:913)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
... 22 more
[2012-09-26 08:45:15,943][WARN ][cluster.action.shard ] [life] sending
failed shard for [prod_threads][0], node[mu_BJdkWRkyYewRElqxzaw], [R],
s[STARTED], reason [Failed to perform [index] on replica, message [Remote
TransportException[Failed to deserialize exception response from stream];
nested: TransportSerializationException[Failed to deserialize exception
response from stream]; nested: StreamCorruptedException[unexpected end of
block data]; ]]
[2012-09-26 08:45:15,943][WARN ][transport.netty ] [life] Message
not fully read (response) for [6224] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOpe
rationAction$4@29a198c9, error [true], resetting
--