Elasticsearch indexing too slow


(Zeeshan Alam) #1

Hi All,

I can't increase the indexing more than 10000 event/second no matter what I do. I am getting around 13000 events per second from kafka in a single logstash instance. I am running 3 Logstash in different machines reading data from same kafka topic.

I have setup a ELK cluster with 3 Logstash reading data from Kafka and sending them to my elastic cluster.

My cluster contains 3 Logstash, 3 Elastic Master Node, 3 Elastic Client node and 50 Elastic Data Node.

All Citrix VM having same configuration of :

Red Hat Linux-7
Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz 6 Cores
32 GB RAM
2 TB spinning media

Logstash Config file :

 output {
    elasticsearch {
      hosts => ["dataNode1:9200","dataNode2:9200","dataNode3:9200" upto "**dataNode50**:9200"]
      index => "logstash-applogs-%{+YYYY.MM.dd}-1"
      workers => 6
      user => "uname"
      password => "pwd"
    }
}

Elasticsearch Data Node's elastcisearch.yml File:

 cluster.name: my-cluster-name
 node.name: node46-data-46
 node.master: false
 node.data: true
 bootstrap.memory_lock: true
 path.data: /apps/dataES1/data
 path.logs: /apps/dataES1/logs
 discovery.zen.ping.unicast.hosts: ["master1","master2","master3"]
 network.host: hostname
 http.port: 9200

The only change that I made in my **jvm.options** file is

-Xms15g
-Xmx15g

System config changes that I did are as follows:

vm.max_map_count=262144

and in /etc/security/limits.conf I added :

elastic       soft    nofile          65536
elastic       hard    nofile          65536
elastic       soft    memlock         unlimited
elastic       hard    memlock         unlimited
elastic       soft    nproc     65536
elastic       hard    nproc     unlimited

Indexing Rate

One of the active data node:

$ sudo iotop -o

Total DISK READ :       0.00 B/s | Total DISK WRITE :     243.29 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     357.09 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 5199 be/3 root        0.00 B/s    3.92 K/s  0.00 %  1.05 % [jbd2/xvdb1-8]
14079 be/4 elkadmin    0.00 B/s   51.01 K/s  0.00 %  0.53 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13936 be/4 elkadmin    0.00 B/s   51.01 K/s  0.00 %  0.39 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13857 be/4 elkadmin    0.00 B/s   58.86 K/s  0.00 %  0.34 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13960 be/4 elkadmin    0.00 B/s   35.32 K/s  0.00 %  0.33 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13964 be/4 elkadmin    0.00 B/s   31.39 K/s  0.00 %  0.27 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
14078 be/4 elkadmin    0.00 B/s   11.77 K/s  0.00 %  0.00 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch

Index Details :

index      shard prirep state docs   store 
2017.01.24-3 3     p  STARTED 5382409  5.4gb
2017.01.24-3 3     r  STARTED 5382409 10.1gb
2017.01.24-3 8     r  STARTED 5383699 10.1gb
2017.01.24-3 8     p  STARTED 5383699 10.2gb
2017.01.24-3 15    r  STARTED 5379509 10.1gb
2017.01.24-3 15    p  STARTED 5379509  5.5gb
2017.01.24-3 5     p  STARTED 5386820 10.3gb
2017.01.24-3 5     r  STARTED 5386820  5.4gb
2017.01.24-3 13    r  STARTED 5386149  6.1gb
2017.01.24-3 13    p  STARTED 5386149  5.4gb
2017.01.24-3 11    r  STARTED 5383620 10.2gb
2017.01.24-3 11    p  STARTED 5383620 10.2gb
2017.01.24-3 19    r  STARTED 5383770  5.7gb
2017.01.24-3 19    p  STARTED 5383770  5.5gb
2017.01.24-3 6     r  STARTED 5389052 10.2gb
2017.01.24-3 6     p  STARTED 5389052  5.8gb
2017.01.24-3 9     p  STARTED 5382316   10gb
2017.01.24-3 9     r  STARTED 5382316  5.6gb
2017.01.24-3 1     p  STARTED 5384850 10.3gb
2017.01.24-3 1     r  STARTED 5384850   10gb
2017.01.24-3 2     p  STARTED 5384110 10.3gb
2017.01.24-3 2     r  STARTED 5384110  5.4gb
2017.01.24-3 14    r  STARTED 5382046  5.4gb
2017.01.24-3 14    p  STARTED 5382046  5.8gb
2017.01.24-3 10    p  STARTED 5383007 10.3gb
2017.01.24-3 10    r  STARTED 5383007  7.7gb
2017.01.24-3 4     r  STARTED 5385397  5.4gb
2017.01.24-3 4     p  STARTED 5385397  5.4gb
2017.01.24-3 7     p  STARTED 5382244  5.5gb
2017.01.24-3 7     r  STARTED 5382244 10.2gb
2017.01.24-3 16    r  STARTED 5381232 10.2gb
2017.01.24-3 16    p  STARTED 5381232  5.5gb
2017.01.24-3 18    r  STARTED 5379792  6.3gb
2017.01.24-3 18    p  STARTED 5379792  5.4gb
2017.01.24-3 12    p  STARTED 5380705  5.4gb
2017.01.24-3 12    r  STARTED 5380705  6.7gb
2017.01.24-3 17    r  STARTED 5382108  5.5gb
2017.01.24-3 17    p  STARTED 5382108  5.8gb
2017.01.24-3 0     r  STARTED 5385285 10.2gb
2017.01.24-3 0     p  STARTED 5385285 10.1gb

I tested the incoming data in logstash by dumping data into a file. I got a file of 290 MB with 377822 lines in 30 seconds. So there is no issue from Kafka as at a given time I am receiving 35000 events per second in my 3 Logstash servers but my Elasticsearch is able to index maximum of 10000 events per second.

Can someone please help me with this issue?


(Christian Dahlqvist) #2

As you are running on VMs I would recommend verifying that the VMs are not over-allocated on the hosts so that they actually have access to the resources you have provisioned. Memory ballooning can cause severe performance problems.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.