Indexing hangs via REST API

[ppearcy]

Hello,
I seem to be getting some hanging when indexing documents. I'm still
playing around to see if I can pin down exactly what is causing the
issue. Also, I am not sure if this is an ES issue, an issue with the
HTTP objects I am using to connect or most likely an OS network/socket
issue, as my indexer is running on Windows, submitting files to ES
running on linux. I've had my fair share of windows socket issues in
the past, caused by horrible default socket connections for the number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to
index. When everything starts up indexing churns away for a while as
it should, but eventually my connection hangs and when this occurs I
see the exception below. I've hit this using a single index thread and
multiple (well, in my case it is actually seperate indexer process to
get around the GIL, but should be the same from ES perspective). This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception below
mean that the socket was closed by the client side before the data was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty ] [Jackhammer] Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
103)
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
461)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
434)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
545)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
754)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
51)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
545)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
540)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
281)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
201)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at
org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

[ppearcy]

Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which points
at fireMessageReceived, so they are different cases. Apologies for any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul ppea...@gmail.com wrote:

Hello,
I seem to be getting some hanging when indexing documents. I'm still
playing around to see if I can pin down exactly what is causing the
issue. Also, I am not sure if this is an ES issue, an issue with the
HTTP objects I am using to connect or most likely an OS network/socket
issue, as my indexer is running on Windows, submitting files to ES
running on linux. I've had my fair share of windows socket issues in
the past, caused by horrible default socket connections for the number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to
index. When everything starts up indexing churns away for a while as
it should, but eventually my connection hangs and when this occurs I
see the exception below. I've hit this using a single index thread and
multiple (well, in my case it is actually seperate indexer process to
get around the GIL, but should be the same from ES perspective). This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception below
mean that the socket was closed by the client side before the data was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty ] [Jackhammer] Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
103)
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
461)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
434)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
545)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
754)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
51)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
545)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
540)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
349)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
281)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
201)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at
org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

[kimchy (Shay Banon)]

Strange... . Can you netstat and see if the number of open sockets keep
getting bigger? Might not use HTTP keep alive and you are running out of
sockets on windows (not sure what happens in windows in this case...)

-shay.banon

On Wed, Aug 4, 2010 at 9:52 PM, Paul ppearcy@gmail.com wrote:

Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which points
at fireMessageReceived, so they are different cases. Apologies for any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul ppea...@gmail.com wrote:

Hello,
I seem to be getting some hanging when indexing documents. I'm still
playing around to see if I can pin down exactly what is causing the
issue. Also, I am not sure if this is an ES issue, an issue with the
HTTP objects I am using to connect or most likely an OS network/socket
issue, as my indexer is running on Windows, submitting files to ES
running on linux. I've had my fair share of windows socket issues in
the past, caused by horrible default socket connections for the number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to
index. When everything starts up indexing churns away for a while as
it should, but eventually my connection hangs and when this occurs I
see the exception below. I've hit this using a single index thread and
multiple (well, in my case it is actually seperate indexer process to
get around the GIL, but should be the same from ES perspective). This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception below
mean that the socket was closed by the client side before the data was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty ] [Jackhammer] Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:

103. at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:

68. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:

81. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

198. at

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

107. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:

461. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:

434. at

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:

80. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at
     org.elasticsearch.common.netty.channel.DefaultChannelPipeline
     $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
546. at

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:

51. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

540. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

274. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

261. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:

349. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:

281. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:

201. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:

108. at

org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

46. at
    

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

1110. at java.util.concurrent.ThreadPoolExecutor
      $Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)

[ppearcy]

When Windows hits its max ephemeral ports it will fail on new attempts
and nothing will ever make it over the wire.

Just to be safe, I applied the Windows registry settings here:

IBM Developer

However, no luck. I did confirm via netstat that keep-alive as my
ephemeral port numbers aren't changing between documents and there
aren't any ES sockets in the WAIT_CLOSE state.

I just hit the condition in the multithreaded case (3 threads) and at
the time on two of the threads and my machine only has a few hundred
sockets active. When I brought down ES, the hung connections were
freed and my indexers started trying to process.

I am going to see if I can get a sniffer trace to see exactly what is
occurring over the wire. I may also play around with running all
processes on linux to avoid potential Windows gotchas.

I think using a HTTP timeout will allow me to work around whatever the
issue is. I have a hunch though that Netty or ES is not properly
closing the socket in this error case.

Thanks,
Paul

On Aug 4, 1:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Strange... . Can you netstat and see if the number of open sockets keep
getting bigger? Might not use HTTP keep alive and you are running out of
sockets on windows (not sure what happens in windows in this case...)

-shay.banon

On Wed, Aug 4, 2010 at 9:52 PM, Paul ppea...@gmail.com wrote:

Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which points
at fireMessageReceived, so they are different cases. Apologies for any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul ppea...@gmail.com wrote:

Hello,
I seem to be getting some hanging when indexing documents. I'm still
playing around to see if I can pin down exactly what is causing the
issue. Also, I am not sure if this is an ES issue, an issue with the
HTTP objects I am using to connect or most likely an OS network/socket
issue, as my indexer is running on Windows, submitting files to ES
running on linux. I've had my fair share of windows socket issues in
the past, caused by horrible default socket connections for the number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to
index. When everything starts up indexing churns away for a while as
it should, but eventually my connection hangs and when this occurs I
see the exception below. I've hit this using a single index thread and
multiple (well, in my case it is actually seperate indexer process to
get around the GIL, but should be the same from ES perspective). This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception below
mean that the socket was closed by the client side before the data was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty ] [Jackhammer] Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:

103. at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:

68. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:

81. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

198. at

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

107. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:

461. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:

434. at

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:

80. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at
     org.elasticsearch.common.netty.channel.DefaultChannelPipeline
     $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
546. at

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:

51. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

540. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

274. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

261. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:

349. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:

281. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:

201. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:

108. at

org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

46. at
    

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

1110. at java.util.concurrent.ThreadPoolExecutor
      $Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)

[kimchy (Shay Banon)]

It can either be that, or a problem in elasticsearch and it does not send
back a response for some reason. I suggest you try the second option, try
and run everything on the linux box and see if it happens there?

-shay.banon

On Wed, Aug 4, 2010 at 11:03 PM, Paul ppearcy@gmail.com wrote:

When Windows hits its max ephemeral ports it will fail on new attempts
and nothing will ever make it over the wire.

Just to be safe, I applied the Windows registry settings here:
http://www.ibm.com/developerworks/lotus/library/quickr-tuning/

However, no luck. I did confirm via netstat that keep-alive as my
ephemeral port numbers aren't changing between documents and there
aren't any ES sockets in the WAIT_CLOSE state.

I just hit the condition in the multithreaded case (3 threads) and at
the time on two of the threads and my machine only has a few hundred
sockets active. When I brought down ES, the hung connections were
freed and my indexers started trying to process.

I am going to see if I can get a sniffer trace to see exactly what is
occurring over the wire. I may also play around with running all
processes on linux to avoid potential Windows gotchas.

I think using a HTTP timeout will allow me to work around whatever the
issue is. I have a hunch though that Netty or ES is not properly
closing the socket in this error case.

Thanks,
Paul

On Aug 4, 1:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Strange... . Can you netstat and see if the number of open sockets keep
getting bigger? Might not use HTTP keep alive and you are running out of
sockets on windows (not sure what happens in windows in this case...)

-shay.banon

On Wed, Aug 4, 2010 at 9:52 PM, Paul ppea...@gmail.com wrote:

Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which points
at fireMessageReceived, so they are different cases. Apologies for any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul ppea...@gmail.com wrote:

Hello,
I seem to be getting some hanging when indexing documents. I'm
still
playing around to see if I can pin down exactly what is causing the
issue. Also, I am not sure if this is an ES issue, an issue with the
HTTP objects I am using to connect or most likely an OS
network/socket
issue, as my indexer is running on Windows, submitting files to ES
running on linux. I've had my fair share of windows socket issues in
the past, caused by horrible default socket connections for the
number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to
index. When everything starts up indexing churns away for a while as
it should, but eventually my connection hangs and when this occurs I
see the exception below. I've hit this using a single index thread
and
multiple (well, in my case it is actually seperate indexer process to
get around the GIL, but should be the same from ES perspective). This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception below
mean that the socket was closed by the client side before the data
was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty ] [Jackhammer] Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:

103. at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:

68. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:

81. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

198. at

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

107. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:

461. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:

434. at

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:

80. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at
     org.elasticsearch.common.netty.channel.DefaultChannelPipeline

$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:

754. at

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:

51. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

540. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

274. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

261. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:

349. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:

281. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:

201. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:

108. at

org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

46. at
    

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

1110. at java.util.concurrent.ThreadPoolExecutor
      $Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)

[ppearcy]

OK, got to the bottom of it... Major bug on my part and minor bug on
the part of ES. I'm throwing a lot of different types of data at the
system and should have checked if this was occurring on a specific
type.

It appears that I had a newline in the path that I was putting indexed
data to. In this case, it appears that ES fails to write an error back
into the connection and a hang occurs.

I've opened an issue for it and included a test to reproduce:

REST API hang on a bogus call

Thanks!

On Aug 4, 3:14 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

It can either be that, or a problem in elasticsearch and it does not send
back a response for some reason. I suggest you try the second option, try
and run everything on the linux box and see if it happens there?

-shay.banon

On Wed, Aug 4, 2010 at 11:03 PM, Paul ppea...@gmail.com wrote:

When Windows hits its max ephemeral ports it will fail on new attempts
and nothing will ever make it over the wire.

Just to be safe, I applied the Windows registry settings here:
IBM Developer

However, no luck. I did confirm via netstat that keep-alive as my
ephemeral port numbers aren't changing between documents and there
aren't any ES sockets in the WAIT_CLOSE state.

I just hit the condition in the multithreaded case (3 threads) and at
the time on two of the threads and my machine only has a few hundred
sockets active. When I brought down ES, the hung connections were
freed and my indexers started trying to process.

I am going to see if I can get a sniffer trace to see exactly what is
occurring over the wire. I may also play around with running all
processes on linux to avoid potential Windows gotchas.

I think using a HTTP timeout will allow me to work around whatever the
issue is. I have a hunch though that Netty or ES is not properly
closing the socket in this error case.

Thanks,
Paul

On Aug 4, 1:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Strange... . Can you netstat and see if the number of open sockets keep
getting bigger? Might not use HTTP keep alive and you are running out of
sockets on windows (not sure what happens in windows in this case...)

-shay.banon

On Wed, Aug 4, 2010 at 9:52 PM, Paul ppea...@gmail.com wrote:

Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which points
at fireMessageReceived, so they are different cases. Apologies for any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul ppea...@gmail.com wrote:

Hello,
I seem to be getting some hanging when indexing documents. I'm
still
playing around to see if I can pin down exactly what is causing the
issue. Also, I am not sure if this is an ES issue, an issue with the
HTTP objects I am using to connect or most likely an OS
network/socket
issue, as my indexer is running on Windows, submitting files to ES
running on linux. I've had my fair share of windows socket issues in
the past, caused by horrible default socket connections for the
number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to
index. When everything starts up indexing churns away for a while as
it should, but eventually my connection hangs and when this occurs I
see the exception below. I've hit this using a single index thread
and
multiple (well, in my case it is actually seperate indexer process to
get around the GIL, but should be the same from ES perspective). This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception below
mean that the socket was closed by the client side before the data
was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty ] [Jackhammer] Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:

103. at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:

68. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:

81. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

198. at

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

107. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:

461. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:

434. at

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:

80. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at
     org.elasticsearch.common.netty.channel.DefaultChannelPipeline

$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:

754. at

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:

51. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

540. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

274. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

261. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:

349. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:

281. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:

201. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:

108. at

org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

46. at
    

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

1110. at java.util.concurrent.ThreadPoolExecutor
      $Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)

[kimchy (Shay Banon)]

great, thanks for the work!. I will look at the case and fix it.

On Thu, Aug 5, 2010 at 7:08 AM, Paul ppearcy@gmail.com wrote:

OK, got to the bottom of it... Major bug on my part and minor bug on
the part of ES. I'm throwing a lot of different types of data at the
system and should have checked if this was occurring on a specific
type.

It appears that I had a newline in the path that I was putting indexed
data to. In this case, it appears that ES fails to write an error back
into the connection and a hang occurs.

I've opened an issue for it and included a test to reproduce:

REST API hang on a bogus call · Issue #297 · elastic/elasticsearch · GitHub

Thanks!

On Aug 4, 3:14 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

It can either be that, or a problem in elasticsearch and it does not send
back a response for some reason. I suggest you try the second option, try
and run everything on the linux box and see if it happens there?

-shay.banon

On Wed, Aug 4, 2010 at 11:03 PM, Paul ppea...@gmail.com wrote:

When Windows hits its max ephemeral ports it will fail on new attempts
and nothing will ever make it over the wire.

Just to be safe, I applied the Windows registry settings here:
IBM Developer

However, no luck. I did confirm via netstat that keep-alive as my
ephemeral port numbers aren't changing between documents and there
aren't any ES sockets in the WAIT_CLOSE state.

I just hit the condition in the multithreaded case (3 threads) and at
the time on two of the threads and my machine only has a few hundred
sockets active. When I brought down ES, the hung connections were
freed and my indexers started trying to process.

I am going to see if I can get a sniffer trace to see exactly what is
occurring over the wire. I may also play around with running all
processes on linux to avoid potential Windows gotchas.

I think using a HTTP timeout will allow me to work around whatever the
issue is. I have a hunch though that Netty or ES is not properly
closing the socket in this error case.

Thanks,
Paul

On Aug 4, 1:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Strange... . Can you netstat and see if the number of open sockets
keep
getting bigger? Might not use HTTP keep alive and you are running out
of
sockets on windows (not sure what happens in windows in this case...)

-shay.banon

On Wed, Aug 4, 2010 at 9:52 PM, Paul ppea...@gmail.com wrote:

Actually, the exception I get when I kill my indexer refers to
fireChannelDisconnected vs the exception on index hangs which
points
at fireMessageReceived, so they are different cases. Apologies for
any
confusion.

Thanks

On Aug 4, 12:41 pm, Paul ppea...@gmail.com wrote:

Hello,
I seem to be getting some hanging when indexing documents. I'm
still
playing around to see if I can pin down exactly what is causing
the
issue. Also, I am not sure if this is an ES issue, an issue with
the
HTTP objects I am using to connect or most likely an OS
network/socket
issue, as my indexer is running on Windows, submitting files to
ES
running on linux. I've had my fair share of windows socket issues
in
the past, caused by horrible default socket connections for the
number
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch
to
index. When everything starts up indexing churns away for a while
as
it should, but eventually my connection hangs and when this
occurs I
see the exception below. I've hit this using a single index
thread
and
multiple (well, in my case it is actually seperate indexer
process to
get around the GIL, but should be the same from ES perspective).
This
is the same exception that I receive when I kill my indexer while
there is an open socket performing an index. Does the exception
below
mean that the socket was closed by the client side before the
data
was
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,
Paul

[18:01:47,706][INFO ][node ] [Jackhammer]
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started
[18:17:18,553][WARN ][http.netty ] [Jackhammer]
Caught
exception while handling client http traffic
java.lang.IllegalArgumentException: empty text
at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:

103. at

org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:

68. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:

81. at
    

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

198. at

org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:

107. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:

461. at

org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:

434. at

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:

80. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at
     org.elasticsearch.common.netty.channel.DefaultChannelPipeline

$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:

754. at

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:

51. at
    

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

545. at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:

540. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

274. at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:

261. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:

349. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:

281. at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:

201. at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:

108. at

org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:

46. at
    

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

1110. at java.util.concurrent.ThreadPoolExecutor
      $Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)