# Elasticsearch indexing too slow **URL:** https://discuss.elastic.co/t/elasticsearch-indexing-too-slow/72934 **Category:** Elasticsearch **Created:** [January 26, 2017, 5:45pm UTC](https://discuss.elastic.co/t/elasticsearch-indexing-too-slow/72934 "2017-01-26T17:45:18Z") **Posts on this page:** 3 **Page:** 1
### Author: ![Zeeshan\_Alam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zeeshan_alam/32/94298_2.png) [@Zeeshan\_Alam](https://discuss.elastic.co/u/Zeeshan_Alam) #### Post date: [January 26, 2017, 5:45pm UTC](https://discuss.elastic.co/t/elasticsearch-indexing-too-slow/72934/1 "2017-01-26T17:45:18Z")
Hi All, I can't increase the indexing more than 10000 event/second no matter what I do. I am getting around 13000 events per second from kafka in a single logstash instance. I am running 3 Logstash in different machines reading data from same kafka topic. I have setup a ELK cluster with 3 Logstash reading data from Kafka and sending them to my elastic cluster. My cluster contains 3 Logstash, 3 Elastic Master Node, 3 Elastic Client node and 50 Elastic Data Node. All Citrix VM having same configuration of : > Red Hat Linux-7 > Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz 6 Cores > 32 GB RAM > 2 TB spinning media **Logstash Config file :** ``` output { elasticsearch { hosts => ["dataNode1:9200","dataNode2:9200","dataNode3:9200" upto " **dataNode50** :9200"] index => "logstash-applogs-%{+YYYY.MM.dd}-1" workers => 6 user => "uname" password => "pwd" } } ``` Elasticsearch Data Node's **elastcisearch.yml** File: ``` cluster.name: my-cluster-name node.name: node46-data-46 node.master: false node.data: true bootstrap.memory_lock: true path.data: /apps/dataES1/data path.logs: /apps/dataES1/logs discovery.zen.ping.unicast.hosts: ["master1","master2","master3"] network.host: hostname http.port: 9200 The only change that I made in my **jvm.options** file is -Xms15g -Xmx15g ``` System config changes that I did are as follows: `vm.max_map_count=262144` and in /etc/security/limits.conf I added : ``` elastic soft nofile 65536 elastic hard nofile 65536 elastic soft memlock unlimited elastic hard memlock unlimited elastic soft nproc 65536 elastic hard nproc unlimited ``` **Indexing Rate** ![](https://us1.discourse-cdn.com/elastic/original/2X/8/878f69e2ee4df02c00dc6cc96ebdd94b57d4ab1c.png) ![](https://us1.discourse-cdn.com/elastic/original/2X/4/4e0a46bc834776af00e319389f15bc0b5618d288.png) **One of the active data node:** _$ sudo iotop -o_ ``` Total DISK READ : 0.00 B/s | Total DISK WRITE : 243.29 K/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 357.09 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 5199 be/3 root 0.00 B/s 3.92 K/s 0.00 % 1.05 % [jbd2/xvdb1-8] 14079 be/4 elkadmin 0.00 B/s 51.01 K/s 0.00 % 0.53 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch 13936 be/4 elkadmin 0.00 B/s 51.01 K/s 0.00 % 0.39 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch 13857 be/4 elkadmin 0.00 B/s 58.86 K/s 0.00 % 0.34 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch 13960 be/4 elkadmin 0.00 B/s 35.32 K/s 0.00 % 0.33 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch 13964 be/4 elkadmin 0.00 B/s 31.39 K/s 0.00 % 0.27 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch 14078 be/4 elkadmin 0.00 B/s 11.77 K/s 0.00 % 0.00 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch ``` ![](https://us1.discourse-cdn.com/elastic/original/2X/0/08a596dff1b399e82e6706c89a1a9fc60663d2de.png) **Index Details :** ``` index shard prirep state docs store 2017.01.24-3 3 p STARTED 5382409 5.4gb 2017.01.24-3 3 r STARTED 5382409 10.1gb 2017.01.24-3 8 r STARTED 5383699 10.1gb 2017.01.24-3 8 p STARTED 5383699 10.2gb 2017.01.24-3 15 r STARTED 5379509 10.1gb 2017.01.24-3 15 p STARTED 5379509 5.5gb 2017.01.24-3 5 p STARTED 5386820 10.3gb 2017.01.24-3 5 r STARTED 5386820 5.4gb 2017.01.24-3 13 r STARTED 5386149 6.1gb 2017.01.24-3 13 p STARTED 5386149 5.4gb 2017.01.24-3 11 r STARTED 5383620 10.2gb 2017.01.24-3 11 p STARTED 5383620 10.2gb 2017.01.24-3 19 r STARTED 5383770 5.7gb 2017.01.24-3 19 p STARTED 5383770 5.5gb 2017.01.24-3 6 r STARTED 5389052 10.2gb 2017.01.24-3 6 p STARTED 5389052 5.8gb 2017.01.24-3 9 p STARTED 5382316 10gb 2017.01.24-3 9 r STARTED 5382316 5.6gb 2017.01.24-3 1 p STARTED 5384850 10.3gb 2017.01.24-3 1 r STARTED 5384850 10gb 2017.01.24-3 2 p STARTED 5384110 10.3gb 2017.01.24-3 2 r STARTED 5384110 5.4gb 2017.01.24-3 14 r STARTED 5382046 5.4gb 2017.01.24-3 14 p STARTED 5382046 5.8gb 2017.01.24-3 10 p STARTED 5383007 10.3gb 2017.01.24-3 10 r STARTED 5383007 7.7gb 2017.01.24-3 4 r STARTED 5385397 5.4gb 2017.01.24-3 4 p STARTED 5385397 5.4gb 2017.01.24-3 7 p STARTED 5382244 5.5gb 2017.01.24-3 7 r STARTED 5382244 10.2gb 2017.01.24-3 16 r STARTED 5381232 10.2gb 2017.01.24-3 16 p STARTED 5381232 5.5gb 2017.01.24-3 18 r STARTED 5379792 6.3gb 2017.01.24-3 18 p STARTED 5379792 5.4gb 2017.01.24-3 12 p STARTED 5380705 5.4gb 2017.01.24-3 12 r STARTED 5380705 6.7gb 2017.01.24-3 17 r STARTED 5382108 5.5gb 2017.01.24-3 17 p STARTED 5382108 5.8gb 2017.01.24-3 0 r STARTED 5385285 10.2gb 2017.01.24-3 0 p STARTED 5385285 10.1gb ``` I tested the incoming data in logstash by dumping data into a file. I got a file of 290 MB with 377822 lines in 30 seconds. So there is no issue from Kafka as at a given time I am receiving 35000 events per second in my 3 Logstash servers but my Elasticsearch is able to index maximum of 10000 events per second. Can someone please help me with this issue? ---
### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist) #### Post date: [January 28, 2017, 8:43am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-too-slow/72934/2 "2017-01-28T08:43:51Z")
As you are running on VMs I would recommend verifying that the VMs are not over-allocated on the hosts so that they actually have access to the resources you have provisioned. Memory ballooning can cause severe performance problems. ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [February 25, 2017, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-indexing-too-slow/72934/3 "2017-02-25T08:44:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.