Upgrade to 2.0 breaks Kibana and Logstash


#1

Hi all.

I just tried to upgrade my elasticsearch from 1.7 to 2.0 on a non-prod system. I have not upgraded logstash or Kibana. Kibana (1.4.2) just stopped working with an error "
Error
Discover: An error occurred with your request. Reset your inputs and try again." And when I start logstash (1.5.4), the elasticsearch.log is filled with below exceptions:

[2015-10-10 16:36:17,099][WARN ][org.elasticsearch.transport.netty] [S'ym] exception caught on transport layer [[id: 0x2d7d20c7, /127.0.0.1:55205 => /127.0.0.1:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [2079], action [internal:discovery/zen/unicast_gte_1_4], readerIndex [59] vs expected [215]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:120)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I could not find any documentation saying, that I have to upgrade kibana and logstash as well, but do I?


(Mark Walkom) #2

You need to use KB 4.2 if you want to use ES 2.0, 4.1 is not compatible.

LS should still work though, what protocol are you using to talk to it?


(Magnus Bäck) #3

[2015-10-10 16:36:17,099][WARN ][org.elasticsearch.transport.netty] [S'ym] exception caught on transport layer [[id: 0x2d7d20c7, /127.0.0.1:55205 => /127.0.0.1:9300]], closing connection

You should not expect Logstash 1.5 to work with ES 2.0 unless you use the HTTP protocol (which will be the default in Logstash 2.0).


#4

Aha okay. Will look into that then.


#5

As far as I can read, the logstash-forwarder only supports the lumberjack protocol. How can I forward logs via HTTP?


(Magnus Bäck) #6

LSF can continue to submit logs via Lumberjack to Logstash, which then sends them to ES via HTTP.


#7

Aha, that way around. Thanks


#8

Hi Magnus,

I have ES 2.0 and LS 1.5.4 and my output to elasticsearch is thru http.
elasticsearch { protocol => "http" host => "10.0.0.35:9200" index => "logstash-%{+YYYY.MM.dd}"}

The new index will be created but in red status and no data is being inserted.
red open logstash-2015.11.05 5 1

Here are some errors in logstash.log
{:timestamp=>"2015-11-05T06:17:43.074000-0500", :message=>"retrying failed action with response code: 503", :level=>:warn}

What could be wrong?


(Magnus Bäck) #9

@rsjavier, please start a new thread for your unrelated question.


#10

OK, I will create one but I upgraded also my ES 1.4.2 to ES 2.0 when I encounter the error.

Thanks!


(system) #11