Error connecting to running ES server - Message not fully read

Elastic Stack Elasticsearch
1

Both java client and server are 0.19.8

This was running just before the latest deployment.

Some other jar dependencies were updated and I see this error just
before the stacktrace below:

log4j:ERROR Error occured while converting date.
java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:345)
at java.lang.StringBuffer.getChars(StringBuffer.java:218)
at org.apache.log4j.helpers.ISO8601DateFormat.format(ISO8601DateFormat.java:130)
at java.text.DateFormat.format(DateFormat.java:333)
at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:443)
at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.warn(Category.java:1043)
at org.elasticsearch.common.logging.log4j.Log4jESLogger.internalWarn(Log4jESLogger.java:114)
at org.elasticsearch.common.logging.support.AbstractESLogger.warn(AbstractESLogger.java:100)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:267)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:154)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:563)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:91)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:373)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:247)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

Here's the exception:

WARN [org.elasticsearch.transport.netty] - <[Day, Wilbur] Message not
fully read (response) for [1] handler
future(org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler$1@1035b2a),
error [false], resetting>
log4j:ERROR Error occured while converting date.
java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:345)
at java.lang.StringBuffer.getChars(StringBuffer.java:218)
at org.apache.log4j.helpers.ISO8601DateFormat.format(ISO8601DateFormat.java:130)
at java.text.DateFormat.format(DateFormat.java:333)
at org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:443)
at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.info(Category.java:683)
at org.elasticsearch.common.logging.log4j.Log4jESLogger.internalInfo(Log4jESLogger.java:109)
at org.elasticsearch.common.logging.support.AbstractESLogger.info(AbstractESLogger.java:90)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:327)
at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:284)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
INFO [org.elasticsearch.client.transport] - <[Day, Wilbur] failed to
get node info for [#transport#-1][inet[localhost/127.0.0.1:9300]],
disconnecting...>
org.elasticsearch.transport.RemoteTransportException: Failed to
deserialize response of type
[org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse]
Caused by: org.elasticsearch.transport.TransportSerializationException:
Failed to deserialize response of type
[org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse]
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:282)
at org.elasticsearch.transport.netty.MessageChannelHandler.process(MessageChannelHandler.java:259)
at org.elasticsearch.transport.netty.MessageChannelHandler.callDecode(MessageChannelHandler.java:154)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:103)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:563)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:91)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:373)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:247)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.NullPointerException
at org.elasticsearch.common.transport.TransportAddressSerializers.addressFromStream(TransportAddressSerializers.java:69)
at org.elasticsearch.cluster.node.DiscoveryNode.readFrom(DiscoveryNode.java:234)
at org.elasticsearch.cluster.node.DiscoveryNode.readNode(DiscoveryNode.java:226)
at org.elasticsearch.action.support.nodes.NodeOperationResponse.readFrom(NodeOperationResponse.java:61)
at org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readFrom(NodeInfo.java:245)
at org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readNodeInfo(NodeInfo.java:239)
at org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse.readFrom(NodesInfoResponse.java:53)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:280)
... 17 more

--

2

Which jars were updated? I've seen issues before when different
versions of Lucene were in the classpath, but not the errors you
listed.

--
Ivan

On Tue, Aug 21, 2012 at 7:15 AM, Warner Onstine warnero@gmail.com wrote:

Some other jar dependencies were updated and I see this error just
before the stacktrace below:

--

3

Some other libraries were added or updated, but I don't think any of
them should be interfering with elasticsearch. Same version of lucene
we were using prior to this 3.6.0.

-warner

On Tue, Aug 21, 2012 at 1:21 PM, Ivan Brusic ivan@brusic.com wrote:

Which jars were updated? I've seen issues before when different
versions of Lucene were in the classpath, but not the errors you
listed.

--
Ivan

On Tue, Aug 21, 2012 at 7:15 AM, Warner Onstine warnero@gmail.com wrote:

Some other jar dependencies were updated and I see this error just
before the stacktrace below:

--

--

4

Looking again I've rolled back the only other library change and we're
still getting this error on startup, anyone have any ideas?

-warner

On Tue, Aug 21, 2012 at 1:55 PM, Warner Onstine warnero@gmail.com wrote:

Some other libraries were added or updated, but I don't think any of
them should be interfering with elasticsearch. Same version of lucene
we were using prior to this 3.6.0.

-warner

On Tue, Aug 21, 2012 at 1:21 PM, Ivan Brusic ivan@brusic.com wrote:

Which jars were updated? I've seen issues before when different
versions of Lucene were in the classpath, but not the errors you
listed.

--
Ivan

On Tue, Aug 21, 2012 at 7:15 AM, Warner Onstine warnero@gmail.com wrote:

Some other jar dependencies were updated and I see this error just
before the stacktrace below:

--

--

5

Looks like our index properties weren't being read properly so it
wasn't connecting to our cluster, not sure why that would be causing
the error we saw. Once we fixed that though our app was able to start
correctly.

-warner

On Tue, Aug 21, 2012 at 8:20 PM, Warner Onstine warnero@gmail.com wrote:

Looking again I've rolled back the only other library change and we're
still getting this error on startup, anyone have any ideas?

-warner

On Tue, Aug 21, 2012 at 1:55 PM, Warner Onstine warnero@gmail.com wrote:

Some other libraries were added or updated, but I don't think any of
them should be interfering with elasticsearch. Same version of lucene
we were using prior to this 3.6.0.

-warner

On Tue, Aug 21, 2012 at 1:21 PM, Ivan Brusic ivan@brusic.com wrote:

Which jars were updated? I've seen issues before when different
versions of Lucene were in the classpath, but not the errors you
listed.

--
Ivan

On Tue, Aug 21, 2012 at 7:15 AM, Warner Onstine warnero@gmail.com wrote:

Some other jar dependencies were updated and I see this error just
before the stacktrace below:

--

--