# Extremely slow indexing -- java throwing http excetion errors

**URL:** <https://discuss.elastic.co/t/extremely-slow-indexing-java-throwing-http-excetion-errors/18194>\
**Category:** Elasticsearch\
**Created:** [June 18, 2014, 10:12pm UTC](https://discuss.elastic.co/t/extremely-slow-indexing-java-throwing-http-excetion-errors/18194 "2014-06-18T22:12:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![alekjouharyan](https://avatars.discourse-cdn.com/v4/letter/a/e495f1/32.png) [@alekjouharyan](https://discuss.elastic.co/u/alekjouharyan)\
**Post date:** [June 18, 2014, 10:12pm UTC](https://discuss.elastic.co/t/extremely-slow-indexing-java-throwing-http-excetion-errors/18194/1 "2014-06-18T22:12:54Z")

</div>

Hello all,

So here's the issue, our cluster was previously very underwhelmed as far as  
resource consumption, and after some config changes (see complete config  
below) -- we were able to hike up resource consumption, but are still  
indexing documents at the same sluggish rate of \< 400 docs/second.

Redis and Logstash are definitely not the bottlenecks, and the indexing  
seems to be growing exponentially worse as we pull in more data. We are  
using elasticsearch v 1.1.1.

The java http exception errors would definitely explain the slugishness, as  
there seems to be a socket timeout every second, like clockwork -- but i'm  
at a loss for what could be causing the errors to begin with.

We are running redis,logstash kibana and the es master (no data) on one  
node, and have our elasticsearch data instance on another node. Network  
latency is definitely not so atrocious that it would be an outright  
bottleneck, and data gets to the secondary node fast enough -- but is  
backed up in indexing.

Any help would greatly be appreciated, and I thank you all in advance!

############### ES CONFIG ###############

index.indexing.slowlog.threshold.index.warn: 10s  
index.indexing.slowlog.threshold.index.info: 5s  
index.indexing.slowlog.threshold.index.debug: 2s  
index.indexing.slowlog.threshold.index.trace: 500ms

monitor.jvm.gc.young.warn: 1000ms  
monitor.jvm.gc.young.info: 700ms  
#monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s  
monitor.jvm.gc.old.info: 5s  
#monitor.jvm.gc.old.debug: 2s  
cluster.name: iislog-cluster  
node.name: "VM-ELKIIS"  
discovery.zen.ping.multicast.enabled: true  
discovery.zen.ping.unicast.hosts: ["192.168.6.145"]  
discovery.zen.ping.timeout: 5  
node.master: true  
node.data: false  
index.number\_of\_shards: 10  
index.number\_of\_replicas: 0  
bootstrap.mlockall: true  
index.refresh\_interval: 30  
indices.memory.index\_buffer\_size: 50%  
index.translog.flush\_threshold\_ops: 50000  
index.store.type: mmapfs  
index.store.compress.stored: true

threadpool.search.type: fixed  
threadpool.search.size: 20  
threadpool.search.queue\_size: 100

threadpool.index.type: fixed  
threadpool.index.size: 20  
threadpool.index.queue\_size: 100

######################## JAVA ERRORS IN ES LOG ###########################

[2014-06-18 09:39:09,565][DEBUG][http.netty] [VM-ELKIIS]  
Caught exception while handling client http traffic, closing connection  
[id: 0x7561184c, /192.168.6.3:6206 =\> /192.168.6.21:9200]  
java.io.IOException: Connection reset by peer  
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)  
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)  
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)  
at sun.nio.ch.IOUtil.read(IOUtil.java:192)  
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:744)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/95e3bc66-b403-4844-a798-da0f25141ca6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/95e3bc66-b403-4844-a798-da0f25141ca6%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 20, 2014, 6:58am UTC](https://discuss.elastic.co/t/extremely-slow-indexing-java-throwing-http-excetion-errors/18194/2 "2014-06-20T06:58:23Z")

</div>

Hey.

judging from the exception this looks like an unstable network connection?  
Are you using persistent HTTP connections? Pinging the nodes by each other  
is not a problem I guess?

--Alex

On Thu, Jun 19, 2014 at 12:12 AM, [alekjouharyan@gmail.com](mailto:alekjouharyan@gmail.com) wrote:

> Hello all,
> 
> So here's the issue, our cluster was previously very underwhelmed as far  
> as resource consumption, and after some config changes (see complete config  
> below) -- we were able to hike up resource consumption, but are still  
> indexing documents at the same sluggish rate of \< 400 docs/second.
> 
> Redis and Logstash are definitely not the bottlenecks, and the indexing  
> seems to be growing exponentially worse as we pull in more data. We are  
> using elasticsearch v 1.1.1.
> 
> The java http exception errors would definitely explain the slugishness,  
> as there seems to be a socket timeout every second, like clockwork -- but  
> i'm at a loss for what could be causing the errors to begin with.
> 
> We are running redis,logstash kibana and the es master (no data) on one  
> node, and have our elasticsearch data instance on another node. Network  
> latency is definitely not so atrocious that it would be an outright  
> bottleneck, and data gets to the secondary node fast enough -- but is  
> backed up in indexing.
> 
> Any help would greatly be appreciated, and I thank you all in advance!
> 
> ############### ES CONFIG ###############
> 
> index.indexing.slowlog.threshold.index.warn: 10s  
> index.indexing.slowlog.threshold.index.info: 5s  
> index.indexing.slowlog.threshold.index.debug: 2s  
> index.indexing.slowlog.threshold.index.trace: 500ms
> 
> monitor.jvm.gc.young.warn: 1000ms  
> monitor.jvm.gc.young.info: 700ms  
> #monitor.jvm.gc.young.debug: 400ms
> 
> monitor.jvm.gc.old.warn: 10s  
> monitor.jvm.gc.old.info: 5s  
> #monitor.jvm.gc.old.debug: 2s  
> cluster.name: iislog-cluster  
> node.name: "VM-ELKIIS"  
> discovery.zen.ping.multicast.enabled: true  
> discovery.zen.ping.unicast.hosts: ["192.168.6.145"]  
> discovery.zen.ping.timeout: 5  
> node.master: true  
> node.data: false  
> index.number\_of\_shards: 10  
> index.number\_of\_replicas: 0  
> bootstrap.mlockall: true  
> index.refresh\_interval: 30  
> indices.memory.index\_buffer\_size: 50%  
> index.translog.flush\_threshold\_ops: 50000  
> index.store.type: mmapfs  
> index.store.compress.stored: true
> 
> threadpool.search.type: fixed  
> threadpool.search.size: 20  
> threadpool.search.queue\_size: 100
> 
> threadpool.index.type: fixed  
> threadpool.index.size: 20  
> threadpool.index.queue\_size: 100
> 
> ######################## JAVA ERRORS IN ES LOG ###########################
> 
> [2014-06-18 09:39:09,565][DEBUG][http.netty] [VM-ELKIIS]  
> Caught exception while handling client http traffic, closing connection  
> [id: 0x7561184c, /192.168.6.3:6206 =\> /192.168.6.21:9200]  
> java.io.IOException: Connection reset by peer  
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)  
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)  
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)  
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)  
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> at java.lang.Thread.run(Thread.java:744)
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/95e3bc66-b403-4844-a798-da0f25141ca6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/95e3bc66-b403-4844-a798-da0f25141ca6%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/95e3bc66-b403-4844-a798-da0f25141ca6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/95e3bc66-b403-4844-a798-da0f25141ca6%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-jK5P8DQWxVPzvcvOsFViFziGwSTnXSbYp689M5wLmMg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-jK5P8DQWxVPzvcvOsFViFziGwSTnXSbYp689M5wLmMg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:21am UTC](https://discuss.elastic.co/t/extremely-slow-indexing-java-throwing-http-excetion-errors/18194/3 "2017-07-06T01:21:01Z")

</div>