High failure rates on REST indexing


(Tim Dysinger-2-3) #1

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:
39)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:
499)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
490)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
427)
at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:
156)
at
org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
49)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
275)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)


(Shay Banon) #2

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysinger@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:
39)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:
499)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
490)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
427)
at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:
156)
at
org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
49)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
275)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device


(Tim Dysinger-2-3) #3

Using this client http://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest Handler.java:
39)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR eceived(ReplayingDecoder.java:
499)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD ecoder.java:
490)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla yingDecoder.java:
427)
at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime outHandler.java:
156)
at
org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel sHandler.java:
49)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker. java:
275)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device


(Shay Banon) #4

Support for HTTP chunks has been added to master:
http://github.com/elasticsearch/elasticsearch/issues/closed#issue/8. You can
get it here: http://www.elasticsearch.com/download/ (with instructions on
how to build it).

-shay.banon

On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger dysinger@gmail.com wrote:

Using this client http://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at

org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest
Handler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR
eceived(ReplayingDecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD
ecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla
yingDecoder.java:

at

org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime
outHandler.java:

at

org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel
sHandler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at

org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.
java:

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at

org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device


(Shay Banon) #5

By the way, it would be interesting the know the numbers you are getting. If
you can, also post the environment you are running on (os, number of CPUs),
the client type, number of clients (concurrent), typical document size and
content, elasticsearch configuration (both node level and index level), cpu
taken by client and by elasticsearch, and anything else you find useful.

-shay.banon

On Sat, Feb 13, 2010 at 11:46 PM, Shay Banon
shay.banon@elasticsearch.comwrote:

Support for HTTP chunks has been added to master:
http://github.com/elasticsearch/elasticsearch/issues/closed#issue/8. You
can get it here: http://www.elasticsearch.com/download/ (with instructions
on how to build it).

-shay.banon

On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger dysinger@gmail.com wrote:

Using this client http://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at

org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest
Handler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR
eceived(ReplayingDecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD
ecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla
yingDecoder.java:

at

org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime
outHandler.java:

at

org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel
sHandler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:

at

org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.
java:

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at

org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device


(Tim Dysinger-2-3) #6

Thanks. It works now.

On Feb 13, 1:46 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Support for HTTP chunks has been added to master:http://github.com/elasticsearch/elasticsearch/issues/closed#issue/8. You can
get it here:http://www.elasticsearch.com/download/(with instructions on
how to build it).

-shay.banon

On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger dysin...@gmail.com wrote:

Using this clienthttp://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at

org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest
Handler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR
eceived(ReplayingDecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD
ecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla
yingDecoder.java:

at

org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime
outHandler.java:

at

org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel
sHandler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at

org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.
java:

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at

org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device


(system) #7