High failure rates on REST indexing

Tim_Dysinger_2_3 · February 13, 2010, 8:00pm

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:
39)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:
499)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
490)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
427)
at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:
156)
at
org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
49)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
275)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

kimchy · February 13, 2010, 8:19pm

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysinger@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:
39)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:
499)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
490)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
427)
at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:
156)
at
org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
49)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
275)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device

Tim_Dysinger_2_3 · February 13, 2010, 8:28pm

Using this client GitHub - francois/rest-client: Simple REST client for Ruby, inspired by microframework syntax for specifying actions.

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest Handler.java:
39)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR eceived(ReplayingDecoder.java:
499)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD ecoder.java:
490)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla yingDecoder.java:
427)
at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime outHandler.java:
156)
at
org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel sHandler.java:
49)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker. java:
275)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device

kimchy · February 13, 2010, 9:46pm

Support for HTTP chunks has been added to master:
Issues · elastic/elasticsearch · GitHub. You can
get it here: http://www.elasticsearch.com/download/ (with instructions on
how to build it).

-shay.banon

On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger dysinger@gmail.com wrote:

Using this client GitHub - francois/rest-client: Simple REST client for Ruby, inspired by microframework syntax for specifying actions.

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at

org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest
Handler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR
eceived(ReplayingDecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD
ecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla
yingDecoder.java:

at

org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime
outHandler.java:

at

org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel
sHandler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at

org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.
java:

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at

org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device

kimchy · February 14, 2010, 12:24am

By the way, it would be interesting the know the numbers you are getting. If
you can, also post the environment you are running on (os, number of CPUs),
the client type, number of clients (concurrent), typical document size and
content, elasticsearch configuration (both node level and index level), cpu
taken by client and by elasticsearch, and anything else you find useful.

-shay.banon

On Sat, Feb 13, 2010 at 11:46 PM, Shay Banon
shay.banon@elasticsearch.comwrote:

Support for HTTP chunks has been added to master:
Issues · elastic/elasticsearch · GitHub. You
can get it here: http://www.elasticsearch.com/download/ (with instructions
on how to build it).

-shay.banon

On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger dysinger@gmail.com wrote:

Using this client GitHub - francois/rest-client: Simple REST client for Ruby, inspired by microframework syntax for specifying actions.

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at

org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest
Handler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR
eceived(ReplayingDecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD
ecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla
yingDecoder.java:

at

org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime
outHandler.java:

at

org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel
sHandler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at

org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.
java:

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at

org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device

Tim_Dysinger_2_3 · February 14, 2010, 1:57am

Thanks. It works now.

On Feb 13, 1:46 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Support for HTTP chunks has been added to master:Issues · elastic/elasticsearch · GitHub. You can
get it here:http://www.elasticsearch.com/download/(with instructions on
how to build it).

-shay.banon

On Sat, Feb 13, 2010 at 10:28 PM, Tim Dysinger dysin...@gmail.com wrote:

Using this clienthttp://github.com/francois/rest-client

I am not explicitly doing chunking but it's possible it's doing this
behind the scenes on larger documents.

On Feb 13, 12:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hey,

This happens basically because there is no support for http chunk yet,
which I guess the http client you use does for large requests.

Is there a chance that you can disable chunking for now? Which http
client are you using?

-shay.banon

On 2/13/10, Tim Dysinger dysin...@gmail.com wrote:

I am rapidly pushing 750k documents to one elasticsearch node
(localhost). About 3% of them fail to push over rest. The json is
fine. I've pushed all these to CouchDB before.

The console just fills up with this

[11:55:54,225][WARN ][http.netty ] [Reilly, Ben] Caught
exception while handling client http trafic
java.lang.ClassCastException:
org.jboss.netty.handler.codec.http.DefaultHttpChunk cannot be cast to
org.jboss.netty.handler.codec.http.HttpRequest
at

org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequest
Handler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
391)
at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageR
eceived(ReplayingDecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingD
ecoder.java:

at

org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Repla
yingDecoder.java:

at

org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTime
outHandler.java:

at

org.elasticsearch.http.netty.OpenChannelsHandler.handleUpstream(OpenChannel
sHandler.java:

at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
345)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
332)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
323)
at

org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.
java:

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
196)
at

org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

--
Sent from my mobile device