# Indexing hangs via REST API

**URL:** <https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187>\
**Category:** Elasticsearch\
**Created:** [August 4, 2010, 6:41pm UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187 "2010-08-04T18:41:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\
**Post date:** [August 4, 2010, 6:41pm UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/1 "2010-08-04T18:41:47Z")

</div>

Hello,  
I seem to be getting some hanging when indexing documents. I'm still  
playing around to see if I can pin down exactly what is causing the  
issue. Also, I am not sure if this is an ES issue, an issue with the  
HTTP objects I am using to connect or most likely an OS network/socket  
issue, as my indexer is running on Windows, submitting files to ES  
running on linux. I've had my fair share of windows socket issues in  
the past, caused by horrible default socket connections for the number  
of ephemeral ports.

I am on master from last night.

I am using the python HTTPConnection object via pyelasticsearch to  
index. When everything starts up indexing churns away for a while as  
it should, but eventually my connection hangs and when this occurs I  
see the exception below. I've hit this using a single index thread and  
multiple (well, in my case it is actually seperate indexer process to  
get around the GIL, but should be the same from ES perspective). This  
is the same exception that I receive when I kill my indexer while  
there is an open socket performing an index. Does the exception below  
mean that the socket was closed by the client side before the data was  
received or while it was flowing over the socket?

Any other thoughts or ideas would be much appreciated.

Thanks,  
Paul

[18:01:47,706][INFO][node] [Jackhammer]  
{elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started  
[18:17:18,553][WARN][http.netty] [Jackhammer] Caught  
exception while handling client http traffic  
java.lang.IllegalArgumentException: empty text  
at  
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:  
103)  
at  
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:  
68)  
at  
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:  
81)  
at  
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:  
198)  
at  
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:  
107)  
at  
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:  
461)  
at  
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:  
434)  
at  
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:  
80)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:  
545)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline  
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:  
754)  
at  
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:  
51)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:  
545)  
at  
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:  
540)  
at  
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:  
274)  
at  
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:  
261)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:  
349)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:  
281)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:  
201)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:  
108)  
at  
org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:  
46)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:  
1110)  
at java.util.concurrent.ThreadPoolExecutor  
$Worker.run(ThreadPoolExecutor.java:603)  
at java.lang.Thread.run(Thread.java:636)

---

<div class="post-metadata">

**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\
**Post date:** [August 4, 2010, 6:52pm UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/2 "2010-08-04T18:52:44Z")

</div>

Actually, the exception I get when I kill my indexer refers to  
fireChannelDisconnected vs the exception on index hangs which points  
at fireMessageReceived, so they are different cases. Apologies for any  
confusion.

Thanks

On Aug 4, 12:41 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:

> Hello,  
> I seem to be getting some hanging when indexing documents. I'm still  
> playing around to see if I can pin down exactly what is causing the  
> issue. Also, I am not sure if this is an ES issue, an issue with the  
> HTTP objects I am using to connect or most likely an OS network/socket  
> issue, as my indexer is running on Windows, submitting files to ES  
> running on linux. I've had my fair share of windows socket issues in  
> the past, caused by horrible default socket connections for the number  
> of ephemeral ports.
> 
> I am on master from last night.
> 
> I am using the python HTTPConnection object via pyelasticsearch to  
> index. When everything starts up indexing churns away for a while as  
> it should, but eventually my connection hangs and when this occurs I  
> see the exception below. I've hit this using a single index thread and  
> multiple (well, in my case it is actually seperate indexer process to  
> get around the GIL, but should be the same from ES perspective). This  
> is the same exception that I receive when I kill my indexer while  
> there is an open socket performing an index. Does the exception below  
> mean that the socket was closed by the client side before the data was  
> received or while it was flowing over the socket?
> 
> Any other thoughts or ideas would be much appreciated.
> 
> Thanks,  
> Paul
> 
> [18:01:47,706][INFO][node] [Jackhammer]  
> {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started  
> [18:17:18,553][WARN][http.netty] [Jackhammer] Caught  
> exception while handling client http traffic  
> java.lang.IllegalArgumentException: empty text  
> at  
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:  
> 103)  
> at  
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:  
> 68)  
> at  
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:  
> 81)  
> at  
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:  
> 198)  
> at  
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:  
> 107)  
> at  
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:  
> 461)  
> at  
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:  
> 434)  
> at  
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:  
> 80)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:  
> 545)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline  
> $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:  
> 754)  
> at  
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:  
> 51)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:  
> 545)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:  
> 540)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:  
> 274)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:  
> 261)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:  
> 349)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:  
> 281)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:  
> 201)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:  
> 108)  
> at  
> org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:  
> 46)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:  
> 1110)  
> at java.util.concurrent.ThreadPoolExecutor  
> $Worker.run(ThreadPoolExecutor.java:603)  
> at java.lang.Thread.run(Thread.java:636)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 4, 2010, 7:13pm UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/3 "2010-08-04T19:13:06Z")

</div>

Strange... . Can you netstat and see if the number of open sockets keep  
getting bigger? Might not use HTTP keep alive and you are running out of  
sockets on windows (not sure what happens in windows in this case...)

-shay.banon

On Wed, Aug 4, 2010 at 9:52 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote:

> Actually, the exception I get when I kill my indexer refers to  
> fireChannelDisconnected vs the exception on index hangs which points  
> at fireMessageReceived, so they are different cases. Apologies for any  
> confusion.
> 
> Thanks
> 
> On Aug 4, 12:41 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> 
> > Hello,  
> > I seem to be getting some hanging when indexing documents. I'm still  
> > playing around to see if I can pin down exactly what is causing the  
> > issue. Also, I am not sure if this is an ES issue, an issue with the  
> > HTTP objects I am using to connect or most likely an OS network/socket  
> > issue, as my indexer is running on Windows, submitting files to ES  
> > running on linux. I've had my fair share of windows socket issues in  
> > the past, caused by horrible default socket connections for the number  
> > of ephemeral ports.
> > 
> > I am on master from last night.
> > 
> > I am using the python HTTPConnection object via pyelasticsearch to  
> > index. When everything starts up indexing churns away for a while as  
> > it should, but eventually my connection hangs and when this occurs I  
> > see the exception below. I've hit this using a single index thread and  
> > multiple (well, in my case it is actually seperate indexer process to  
> > get around the GIL, but should be the same from ES perspective). This  
> > is the same exception that I receive when I kill my indexer while  
> > there is an open socket performing an index. Does the exception below  
> > mean that the socket was closed by the client side before the data was  
> > received or while it was flowing over the socket?
> > 
> > Any other thoughts or ideas would be much appreciated.
> > 
> > Thanks,  
> > Paul
> > 
> > [18:01:47,706][INFO][node] [Jackhammer]  
> > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started  
> > [18:17:18,553][WARN][http.netty] [Jackhammer] Caught  
> > exception while handling client http traffic  
> > java.lang.IllegalArgumentException: empty text  
> > at
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> 
> > 1. 
> > 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> 
> > 1. 
> > 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> 
> > 1. 
> > 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > 1. at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline  
> > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > 2. at
> 
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> 
> > 1. 
> > 
> > ```
> > at
> > 
> > ```
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> 
> > 1. at
> 
> org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> 
> > 1. 
> > 
> > ```
> > at
> > 
> > ```
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 
> > 1. at java.util.concurrent.ThreadPoolExecutor  
> > $Worker.run(ThreadPoolExecutor.java:603)  
> > at java.lang.Thread.run(Thread.java:636)

---

<div class="post-metadata">

**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\
**Post date:** [August 4, 2010, 8:03pm UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/4 "2010-08-04T20:03:46Z")

</div>

When Windows hits its max ephemeral ports it will fail on new attempts  
and nothing will ever make it over the wire.

Just to be safe, I applied the Windows registry settings here:

> **[IBM Developer](https://developer.ibm.com/product-doclinks/)**
>
> IBM Developer is your one-stop location for getting hands-on training and learning in-demand skills on relevant technologies such as generative AI, data science, AI, and open source.

However, no luck. I did confirm via netstat that keep-alive as my  
ephemeral port numbers aren't changing between documents and there  
aren't any ES sockets in the WAIT\_CLOSE state.

I just hit the condition in the multithreaded case (3 threads) and at  
the time on two of the threads and my machine only has a few hundred  
sockets active. When I brought down ES, the hung connections were  
freed and my indexers started trying to process.

I am going to see if I can get a sniffer trace to see exactly what is  
occurring over the wire. I may also play around with running all  
processes on linux to avoid potential Windows gotchas.

I think using a HTTP timeout will allow me to work around whatever the  
issue is. I have a hunch though that Netty or ES is not properly  
closing the socket in this error case.

Thanks,  
Paul

On Aug 4, 1:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Strange... . Can you netstat and see if the number of open sockets keep  
> getting bigger? Might not use HTTP keep alive and you are running out of  
> sockets on windows (not sure what happens in windows in this case...)
> 
> -shay.banon
> 
> On Wed, Aug 4, 2010 at 9:52 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> 
> > Actually, the exception I get when I kill my indexer refers to  
> > fireChannelDisconnected vs the exception on index hangs which points  
> > at fireMessageReceived, so they are different cases. Apologies for any  
> > confusion.
> 
> > Thanks
> 
> > On Aug 4, 12:41 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > 
> > > Hello,  
> > > I seem to be getting some hanging when indexing documents. I'm still  
> > > playing around to see if I can pin down exactly what is causing the  
> > > issue. Also, I am not sure if this is an ES issue, an issue with the  
> > > HTTP objects I am using to connect or most likely an OS network/socket  
> > > issue, as my indexer is running on Windows, submitting files to ES  
> > > running on linux. I've had my fair share of windows socket issues in  
> > > the past, caused by horrible default socket connections for the number  
> > > of ephemeral ports.
> 
> > > I am on master from last night.
> 
> > > I am using the python HTTPConnection object via pyelasticsearch to  
> > > index. When everything starts up indexing churns away for a while as  
> > > it should, but eventually my connection hangs and when this occurs I  
> > > see the exception below. I've hit this using a single index thread and  
> > > multiple (well, in my case it is actually seperate indexer process to  
> > > get around the GIL, but should be the same from ES perspective). This  
> > > is the same exception that I receive when I kill my indexer while  
> > > there is an open socket performing an index. Does the exception below  
> > > mean that the socket was closed by the client side before the data was  
> > > received or while it was flowing over the socket?
> 
> > > Any other thoughts or ideas would be much appreciated.
> 
> > > Thanks,  
> > > Paul
> 
> > > [18:01:47,706][INFO][node] [Jackhammer]  
> > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started  
> > > [18:17:18,553][WARN][http.netty] [Jackhammer] Caught  
> > > exception while handling client http traffic  
> > > java.lang.IllegalArgumentException: empty text  
> > > at
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> > 
> > > 1. 
> > > 
> > > ```
> > > at
> > > 
> > > ```
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> > 
> > > 1. 
> > > 
> > > ```
> > > at
> > > 
> > > ```
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> > 
> > > 1. 
> > > 
> > > ```
> > > at
> > > 
> > > ```
> 
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > 
> > > 1. at  
> > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline  
> > > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > > 2. at
> 
> > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> > 
> > > 1. 
> > > 
> > > ```
> > > at
> > > 
> > > ```
> 
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> > 
> > > 1. at
> 
> > org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > 
> > > 1. 
> > > 
> > > ```
> > > at
> > > 
> > > ```
> 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> > 
> > > 1. at java.util.concurrent.ThreadPoolExecutor  
> > > $Worker.run(ThreadPoolExecutor.java:603)  
> > > at java.lang.Thread.run(Thread.java:636)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 4, 2010, 9:14pm UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/5 "2010-08-04T21:14:19Z")

</div>

It can either be that, or a problem in elasticsearch and it does not send  
back a response for some reason. I suggest you try the second option, try  
and run everything on the linux box and see if it happens there?

-shay.banon

On Wed, Aug 4, 2010 at 11:03 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote:

> When Windows hits its max ephemeral ports it will fail on new attempts  
> and nothing will ever make it over the wire.
> 
> Just to be safe, I applied the Windows registry settings here:  
> [http://www.ibm.com/developerworks/lotus/library/quickr-tuning/](http://www.ibm.com/developerworks/lotus/library/quickr-tuning/)
> 
> However, no luck. I did confirm via netstat that keep-alive as my  
> ephemeral port numbers aren't changing between documents and there  
> aren't any ES sockets in the WAIT\_CLOSE state.
> 
> I just hit the condition in the multithreaded case (3 threads) and at  
> the time on two of the threads and my machine only has a few hundred  
> sockets active. When I brought down ES, the hung connections were  
> freed and my indexers started trying to process.
> 
> I am going to see if I can get a sniffer trace to see exactly what is  
> occurring over the wire. I may also play around with running all  
> processes on linux to avoid potential Windows gotchas.
> 
> I think using a HTTP timeout will allow me to work around whatever the  
> issue is. I have a hunch though that Netty or ES is not properly  
> closing the socket in this error case.
> 
> Thanks,  
> Paul
> 
> On Aug 4, 1:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Strange... . Can you netstat and see if the number of open sockets keep  
> > getting bigger? Might not use HTTP keep alive and you are running out of  
> > sockets on windows (not sure what happens in windows in this case...)
> > 
> > -shay.banon
> > 
> > On Wed, Aug 4, 2010 at 9:52 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > 
> > > Actually, the exception I get when I kill my indexer refers to  
> > > fireChannelDisconnected vs the exception on index hangs which points  
> > > at fireMessageReceived, so they are different cases. Apologies for any  
> > > confusion.
> > 
> > > Thanks
> > 
> > > On Aug 4, 12:41 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > > 
> > > > Hello,  
> > > > I seem to be getting some hanging when indexing documents. I'm  
> > > > still  
> > > > playing around to see if I can pin down exactly what is causing the  
> > > > issue. Also, I am not sure if this is an ES issue, an issue with the  
> > > > HTTP objects I am using to connect or most likely an OS  
> > > > network/socket  
> > > > issue, as my indexer is running on Windows, submitting files to ES  
> > > > running on linux. I've had my fair share of windows socket issues in  
> > > > the past, caused by horrible default socket connections for the  
> > > > number  
> > > > of ephemeral ports.
> > 
> > > > I am on master from last night.
> > 
> > > > I am using the python HTTPConnection object via pyelasticsearch to  
> > > > index. When everything starts up indexing churns away for a while as  
> > > > it should, but eventually my connection hangs and when this occurs I  
> > > > see the exception below. I've hit this using a single index thread  
> > > > and  
> > > > multiple (well, in my case it is actually seperate indexer process to  
> > > > get around the GIL, but should be the same from ES perspective). This  
> > > > is the same exception that I receive when I kill my indexer while  
> > > > there is an open socket performing an index. Does the exception below  
> > > > mean that the socket was closed by the client side before the data  
> > > > was  
> > > > received or while it was flowing over the socket?
> > 
> > > > Any other thoughts or ideas would be much appreciated.
> > 
> > > > Thanks,  
> > > > Paul
> > 
> > > > [18:01:47,706][INFO][node] [Jackhammer]  
> > > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started  
> > > > [18:17:18,553][WARN][http.netty] [Jackhammer] Caught  
> > > > exception while handling client http traffic  
> > > > java.lang.IllegalArgumentException: empty text  
> > > > at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> 
> > > > 1. 
> > > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> 
> > > > 1. 
> > > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> 
> > > > 1. 
> > > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > > > 1. at  
> > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> 
> $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> 
> > > > 1. 
> > > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> 
> > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> 
> > > > 1. 
> > > > 
> > > > ```
> > > > at
> > > > 
> > > > ```
> > 
> > >
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 
> > > > 1. at java.util.concurrent.ThreadPoolExecutor  
> > > > $Worker.run(ThreadPoolExecutor.java:603)  
> > > > at java.lang.Thread.run(Thread.java:636)

---

<div class="post-metadata">

**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\
**Post date:** [August 5, 2010, 4:08am UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/6 "2010-08-05T04:08:28Z")

</div>

OK, got to the bottom of it... Major bug on my part and minor bug on  
the part of ES. I'm throwing a lot of different types of data at the  
system and should have checked if this was occurring on a specific  
type.

It appears that I had a newline in the path that I was putting indexed  
data to. In this case, it appears that ES fails to write an error back  
into the connection and a hang occurs.

I've opened an issue for it and included a test to reproduce:

> <https://github.com/elastic/elasticsearch/issues/297>
>
> Let me know if there are any questions.
> 
> Here is a test in python to reproduce. …No need to have index test or doctype test setup.
> 
> from httplib import HTTPConnection
> 
> path = 'i/test/test/101-DSTENVIRONMENT01\\n-143352270'
> body = '{"blah": "101"}'
> 
> conn = HTTPConnection('dm-adsearchd101', 9200)
> conn.request('PUT', path, body)
> response = conn.getresponse()
> 
> print ('passed')

Thanks!

On Aug 4, 3:14 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> It can either be that, or a problem in elasticsearch and it does not send  
> back a response for some reason. I suggest you try the second option, try  
> and run everything on the linux box and see if it happens there?
> 
> -shay.banon
> 
> On Wed, Aug 4, 2010 at 11:03 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> 
> > When Windows hits its max ephemeral ports it will fail on new attempts  
> > and nothing will ever make it over the wire.
> 
> > Just to be safe, I applied the Windows registry settings here:  
> > [IBM Developer](http://www.ibm.com/developerworks/lotus/library/quickr-tuning/)
> 
> > However, no luck. I did confirm via netstat that keep-alive as my  
> > ephemeral port numbers aren't changing between documents and there  
> > aren't any ES sockets in the WAIT\_CLOSE state.
> 
> > I just hit the condition in the multithreaded case (3 threads) and at  
> > the time on two of the threads and my machine only has a few hundred  
> > sockets active. When I brought down ES, the hung connections were  
> > freed and my indexers started trying to process.
> 
> > I am going to see if I can get a sniffer trace to see exactly what is  
> > occurring over the wire. I may also play around with running all  
> > processes on linux to avoid potential Windows gotchas.
> 
> > I think using a HTTP timeout will allow me to work around whatever the  
> > issue is. I have a hunch though that Netty or ES is not properly  
> > closing the socket in this error case.
> 
> > Thanks,  
> > Paul
> 
> > On Aug 4, 1:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Strange... . Can you netstat and see if the number of open sockets keep  
> > > getting bigger? Might not use HTTP keep alive and you are running out of  
> > > sockets on windows (not sure what happens in windows in this case...)
> 
> > > -shay.banon
> 
> > > On Wed, Aug 4, 2010 at 9:52 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > > 
> > > > Actually, the exception I get when I kill my indexer refers to  
> > > > fireChannelDisconnected vs the exception on index hangs which points  
> > > > at fireMessageReceived, so they are different cases. Apologies for any  
> > > > confusion.
> 
> > > > Thanks
> 
> > > > On Aug 4, 12:41 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > > > 
> > > > > Hello,  
> > > > > I seem to be getting some hanging when indexing documents. I'm  
> > > > > still  
> > > > > playing around to see if I can pin down exactly what is causing the  
> > > > > issue. Also, I am not sure if this is an ES issue, an issue with the  
> > > > > HTTP objects I am using to connect or most likely an OS  
> > > > > network/socket  
> > > > > issue, as my indexer is running on Windows, submitting files to ES  
> > > > > running on linux. I've had my fair share of windows socket issues in  
> > > > > the past, caused by horrible default socket connections for the  
> > > > > number  
> > > > > of ephemeral ports.
> 
> > > > > I am on master from last night.
> 
> > > > > I am using the python HTTPConnection object via pyelasticsearch to  
> > > > > index. When everything starts up indexing churns away for a while as  
> > > > > it should, but eventually my connection hangs and when this occurs I  
> > > > > see the exception below. I've hit this using a single index thread  
> > > > > and  
> > > > > multiple (well, in my case it is actually seperate indexer process to  
> > > > > get around the GIL, but should be the same from ES perspective). This  
> > > > > is the same exception that I receive when I kill my indexer while  
> > > > > there is an open socket performing an index. Does the exception below  
> > > > > mean that the socket was closed by the client side before the data  
> > > > > was  
> > > > > received or while it was flowing over the socket?
> 
> > > > > Any other thoughts or ideas would be much appreciated.
> 
> > > > > Thanks,  
> > > > > Paul
> 
> > > > > [18:01:47,706][INFO][node] [Jackhammer]  
> > > > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started  
> > > > > [18:17:18,553][WARN][http.netty] [Jackhammer] Caught  
> > > > > exception while handling client http traffic  
> > > > > java.lang.IllegalArgumentException: empty text  
> > > > > at
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> > 
> > > > > 1. 
> > > > > 
> > > > > ```
> > > > > at
> > > > > 
> > > > > ```
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> > 
> > > > > 1. 
> > > > > 
> > > > > ```
> > > > > at
> > > > > 
> > > > > ```
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> > 
> > > > > 1. 
> > > > > 
> > > > > ```
> > > > > at
> > > > > 
> > > > > ```
> 
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > 
> > > > > 1. at  
> > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> 
> > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> > 
> > > > > 1. 
> > > > > 
> > > > > ```
> > > > > at
> > > > > 
> > > > > ```
> 
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> > 
> > > > > 1. at
> 
> > org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> > 
> > > > > 1. 
> > > > > 
> > > > > ```
> > > > > at
> > > > > 
> > > > > ```
> 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> > 
> > > > > 1. at java.util.concurrent.ThreadPoolExecutor  
> > > > > $Worker.run(ThreadPoolExecutor.java:603)  
> > > > > at java.lang.Thread.run(Thread.java:636)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 5, 2010, 7:43am UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/7 "2010-08-05T07:43:24Z")

</div>

great, thanks for the work!. I will look at the case and fix it.

On Thu, Aug 5, 2010 at 7:08 AM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote:

> OK, got to the bottom of it... Major bug on my part and minor bug on  
> the part of ES. I'm throwing a lot of different types of data at the  
> system and should have checked if this was occurring on a specific  
> type.
> 
> It appears that I had a newline in the path that I was putting indexed  
> data to. In this case, it appears that ES fails to write an error back  
> into the connection and a hang occurs.
> 
> I've opened an issue for it and included a test to reproduce:
> 
> [REST API hang on a bogus call · Issue #297 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/297)
> 
> Thanks!
> 
> On Aug 4, 3:14 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > It can either be that, or a problem in elasticsearch and it does not send  
> > back a response for some reason. I suggest you try the second option, try  
> > and run everything on the linux box and see if it happens there?
> > 
> > -shay.banon
> > 
> > On Wed, Aug 4, 2010 at 11:03 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > 
> > > When Windows hits its max ephemeral ports it will fail on new attempts  
> > > and nothing will ever make it over the wire.
> > 
> > > Just to be safe, I applied the Windows registry settings here:  
> > > [IBM Developer](http://www.ibm.com/developerworks/lotus/library/quickr-tuning/)
> > 
> > > However, no luck. I did confirm via netstat that keep-alive as my  
> > > ephemeral port numbers aren't changing between documents and there  
> > > aren't any ES sockets in the WAIT\_CLOSE state.
> > 
> > > I just hit the condition in the multithreaded case (3 threads) and at  
> > > the time on two of the threads and my machine only has a few hundred  
> > > sockets active. When I brought down ES, the hung connections were  
> > > freed and my indexers started trying to process.
> > 
> > > I am going to see if I can get a sniffer trace to see exactly what is  
> > > occurring over the wire. I may also play around with running all  
> > > processes on linux to avoid potential Windows gotchas.
> > 
> > > I think using a HTTP timeout will allow me to work around whatever the  
> > > issue is. I have a hunch though that Netty or ES is not properly  
> > > closing the socket in this error case.
> > 
> > > Thanks,  
> > > Paul
> > 
> > > On Aug 4, 1:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > 
> > > > Strange... . Can you netstat and see if the number of open sockets  
> > > > keep  
> > > > getting bigger? Might not use HTTP keep alive and you are running out  
> > > > of  
> > > > sockets on windows (not sure what happens in windows in this case...)
> > 
> > > > -shay.banon
> > 
> > > > On Wed, Aug 4, 2010 at 9:52 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > > > 
> > > > > Actually, the exception I get when I kill my indexer refers to  
> > > > > fireChannelDisconnected vs the exception on index hangs which  
> > > > > points  
> > > > > at fireMessageReceived, so they are different cases. Apologies for  
> > > > > any  
> > > > > confusion.
> > 
> > > > > Thanks
> > 
> > > > > On Aug 4, 12:41 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
> > > > > 
> > > > > > Hello,  
> > > > > > I seem to be getting some hanging when indexing documents. I'm  
> > > > > > still  
> > > > > > playing around to see if I can pin down exactly what is causing  
> > > > > > the  
> > > > > > issue. Also, I am not sure if this is an ES issue, an issue with  
> > > > > > the  
> > > > > > HTTP objects I am using to connect or most likely an OS  
> > > > > > network/socket  
> > > > > > issue, as my indexer is running on Windows, submitting files to  
> > > > > > ES  
> > > > > > running on linux. I've had my fair share of windows socket issues  
> > > > > > in  
> > > > > > the past, caused by horrible default socket connections for the  
> > > > > > number  
> > > > > > of ephemeral ports.
> > 
> > > > > > I am on master from last night.
> > 
> > > > > > I am using the python HTTPConnection object via pyelasticsearch  
> > > > > > to  
> > > > > > index. When everything starts up indexing churns away for a while  
> > > > > > as  
> > > > > > it should, but eventually my connection hangs and when this  
> > > > > > occurs I  
> > > > > > see the exception below. I've hit this using a single index  
> > > > > > thread  
> > > > > > and  
> > > > > > multiple (well, in my case it is actually seperate indexer  
> > > > > > process to  
> > > > > > get around the GIL, but should be the same from ES perspective).  
> > > > > > This  
> > > > > > is the same exception that I receive when I kill my indexer while  
> > > > > > there is an open socket performing an index. Does the exception  
> > > > > > below  
> > > > > > mean that the socket was closed by the client side before the  
> > > > > > data  
> > > > > > was  
> > > > > > received or while it was flowing over the socket?
> > 
> > > > > > Any other thoughts or ideas would be much appreciated.
> > 
> > > > > > Thanks,  
> > > > > > Paul
> > 
> > > > > > [18:01:47,706][INFO][node] [Jackhammer]  
> > > > > > {elasticsearch/0.9.1-SNAPSHOT/2010-08-04T06:38:28}[7850]: started  
> > > > > > [18:17:18,553][WARN][http.netty] [Jackhammer]  
> > > > > > Caught  
> > > > > > exception while handling client http traffic  
> > > > > > java.lang.IllegalArgumentException: empty text  
> > > > > > at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
> 
> > > > > > 1. 
> > > > > > 
> > > > > > ```
> > > > > > at
> > > > > > 
> > > > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
> 
> > > > > > 1. 
> > > > > > 
> > > > > > ```
> > > > > > at
> > > > > > 
> > > > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
> 
> > > > > > 1. 
> > > > > > 
> > > > > > ```
> > > > > > at
> > > > > > 
> > > > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > > > > > 1. at  
> > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline
> > 
> > > $DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
> > > 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
> 
> > > > > > 1. 
> > > > > > 
> > > > > > ```
> > > > > > at
> > > > > > 
> > > > > > ```
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
> 
> > > > > > 1. at
> > 
> > >
> 
> org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
> 
> > > > > > 1. 
> > > > > > 
> > > > > > ```
> > > > > > at
> > > > > > 
> > > > > > ```
> > 
> > >
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 
> > > > > > 1. at java.util.concurrent.ThreadPoolExecutor  
> > > > > > $Worker.run(ThreadPoolExecutor.java:603)  
> > > > > > at java.lang.Thread.run(Thread.java:636)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:21am UTC](https://discuss.elastic.co/t/indexing-hangs-via-rest-api/3187/8 "2017-07-06T04:21:06Z")

</div>