Indexing fails with transport error

Elastic Stack Elasticsearch
1

Hi I`m new to search and indexing.

I am using nutch2.3.1/hbase1.1.5/elastic2.3.0 stack installed in an AWS m4.xlarge instance. Everything running in single-node configurations.

After crawling a website, I can see the content in HBase, but when I try to index that using
./nutch index elasticsearch -all

I get following error in the elasticsearch

[2016-07-06 15:46:01,487][WARN ][transport.netty          ] [Mordred] exception caught on transport layer [[id: 0xc3ad0eda, /127.0.0.1:42834 => /127.0.0.1:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [0], action [cluster/nodes/info], readerIndex [39] vs expected [57]; resetting
	at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:120)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I am not able to proceed as I cannot find any resource that would explain how to resolve the issue. :sweat:

Any help is much appreciated.

Thanks in advance :+1:

2

A guess is that Nutch is sending data using the Transport protocol (port 9300) using a format elasticsearch does not understand.

Either this version of Nutch is not compatible with this elasticsearch version.
Or Nutch should be configured to use REST port (9200)?

I don't know nutch so can't really tell more.

3

Thanks very much, It was mu guess as well, because I am using the latest nutch and hbase version. So the elasticsearch integration might not be up-to-date,

Nutch is using the correct port it seems. checked the config files

I am going to try a stack with more older versions, to see whether I can running, and then build on from there.