Hello all,
So here's the issue, our cluster was previously very underwhelmed as far as
resource consumption, and after some config changes (see complete config
below) -- we were able to hike up resource consumption, but are still
indexing documents at the same sluggish rate of < 400 docs/second.
Redis and Logstash are definitely not the bottlenecks, and the indexing
seems to be growing exponentially worse as we pull in more data. We are
using elasticsearch v 1.1.1.
The java http exception errors would definitely explain the slugishness, as
there seems to be a socket timeout every second, like clockwork -- but i'm
at a loss for what could be causing the errors to begin with.
We are running redis,logstash kibana and the es master (no data) on one
node, and have our elasticsearch data instance on another node. Network
latency is definitely not so atrocious that it would be an outright
bottleneck, and data gets to the secondary node fast enough -- but is
backed up in indexing.
Any help would greatly be appreciated, and I thank you all in advance!
############### ES CONFIG ###############
index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms
monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
#monitor.jvm.gc.young.debug: 400ms
monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
#monitor.jvm.gc.old.debug: 2s
cluster.name: iislog-cluster
node.name: "VM-ELKIIS"
discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.unicast.hosts: ["192.168.6.145"]
discovery.zen.ping.timeout: 5
node.master: true
node.data: false
index.number_of_shards: 10
index.number_of_replicas: 0
bootstrap.mlockall: true
index.refresh_interval: 30
indices.memory.index_buffer_size: 50%
index.translog.flush_threshold_ops: 50000
index.store.type: mmapfs
index.store.compress.stored: true
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 100
threadpool.index.type: fixed
threadpool.index.size: 20
threadpool.index.queue_size: 100
######################## JAVA ERRORS IN ES LOG ###########################
[2014-06-18 09:39:09,565][DEBUG][http.netty ] [VM-ELKIIS]
Caught exception while handling client http traffic, closing connection
[id: 0x7561184c, /192.168.6.3:6206 => /192.168.6.21:9200]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/95e3bc66-b403-4844-a798-da0f25141ca6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.