Using elasticsearch embedded in tomcat 7 where we have custom classloader
that shares spring application beans with many webapps. The API's to these
implementations are in shared but the implementations are in a separate
classloader "ComponentLoader". Our search implementation is loaded from
ComponentLoader where elasticsearch has been promoted as our default search
engine.
Everything works fine on a single node but when in a cluster seeing the
following ClassNotFoundException issue when shards are trying to update on
other nodes.
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.transport.RemoteTransportException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
I thought this might have been related
to https://github.com/elasticsearch/elasticsearch/issues/4634 but that
doesn't seem to be the case.
My next idea is to look at why
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass is
invoking WebappClassLoader instead of our custom ComponentLoader and coerce
it to use ours.
But I wanted to get some opinions on this strategy as I am new to
elasticsearch 
my thanks,
Earle
Full stack trace:
2014-08-05 01:25:26,685 WARN elasticsearch[app02][generic][T#1]
org.elasticsearch.indices.cluster - [app02] [sakai_index][0] failed to
start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[sakai_index][0]: Recovery failed from
[app01][0YtJIFeHSuehUfjjfMgv6A][ip-10-93-162-196][inet[/10.93.162.196:9300]]{local=false}
into
[app02][yshRulD0QjaNyu40Z6EGWQ][ip-10-7-174-145][inet[/10.7.174.145:9300]]{local=false}
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65)
at
org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: Failed to
deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize exception response from stream
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.InvalidClassException: failed to read class descriptor
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1603)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167)
... 23 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.transport.RemoteTransportException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
... 28 more
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17542a40-7b6c-4c31-9e3e-1fdd2fe7f90b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
