Elasticsearch indexing too slow

Elastic Stack Elasticsearch
1

Hi All,

I can't increase the indexing more than 10000 event/second no matter what I do. I am getting around 13000 events per second from kafka in a single logstash instance. I am running 3 Logstash in different machines reading data from same kafka topic.

I have setup a ELK cluster with 3 Logstash reading data from Kafka and sending them to my elastic cluster.

My cluster contains 3 Logstash, 3 Elastic Master Node, 3 Elastic Client node and 50 Elastic Data Node.

All Citrix VM having same configuration of :

Red Hat Linux-7
Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz 6 Cores
32 GB RAM
2 TB spinning media

Logstash Config file :

 output {
    elasticsearch {
      hosts => ["dataNode1:9200","dataNode2:9200","dataNode3:9200" upto "**dataNode50**:9200"]
      index => "logstash-applogs-%{+YYYY.MM.dd}-1"
      workers => 6
      user => "uname"
      password => "pwd"
    }
}

Elasticsearch Data Node's elastcisearch.yml File:

 cluster.name: my-cluster-name
 node.name: node46-data-46
 node.master: false
 node.data: true
 bootstrap.memory_lock: true
 path.data: /apps/dataES1/data
 path.logs: /apps/dataES1/logs
 discovery.zen.ping.unicast.hosts: ["master1","master2","master3"]
 network.host: hostname
 http.port: 9200

The only change that I made in my **jvm.options** file is

-Xms15g
-Xmx15g

System config changes that I did are as follows:

vm.max_map_count=262144

and in /etc/security/limits.conf I added :

elastic       soft    nofile          65536
elastic       hard    nofile          65536
elastic       soft    memlock         unlimited
elastic       hard    memlock         unlimited
elastic       soft    nproc     65536
elastic       hard    nproc     unlimited

Indexing Rate

1581×741 85.5 KB

1413×355 16.5 KB

One of the active data node:

$ sudo iotop -o

Total DISK READ :       0.00 B/s | Total DISK WRITE :     243.29 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     357.09 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 5199 be/3 root        0.00 B/s    3.92 K/s  0.00 %  1.05 % [jbd2/xvdb1-8]
14079 be/4 elkadmin    0.00 B/s   51.01 K/s  0.00 %  0.53 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13936 be/4 elkadmin    0.00 B/s   51.01 K/s  0.00 %  0.39 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13857 be/4 elkadmin    0.00 B/s   58.86 K/s  0.00 %  0.34 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13960 be/4 elkadmin    0.00 B/s   35.32 K/s  0.00 %  0.33 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
13964 be/4 elkadmin    0.00 B/s   31.39 K/s  0.00 %  0.27 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch
14078 be/4 elkadmin    0.00 B/s   11.77 K/s  0.00 %  0.00 % java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSIni~h-5.0.2/lib/* org.elasticsearch.bootstrap.Elasticsearch

1437×713 61.4 KB

Index Details :

index      shard prirep state docs   store 
2017.01.24-3 3     p  STARTED 5382409  5.4gb
2017.01.24-3 3     r  STARTED 5382409 10.1gb
2017.01.24-3 8     r  STARTED 5383699 10.1gb
2017.01.24-3 8     p  STARTED 5383699 10.2gb
2017.01.24-3 15    r  STARTED 5379509 10.1gb
2017.01.24-3 15    p  STARTED 5379509  5.5gb
2017.01.24-3 5     p  STARTED 5386820 10.3gb
2017.01.24-3 5     r  STARTED 5386820  5.4gb
2017.01.24-3 13    r  STARTED 5386149  6.1gb
2017.01.24-3 13    p  STARTED 5386149  5.4gb
2017.01.24-3 11    r  STARTED 5383620 10.2gb
2017.01.24-3 11    p  STARTED 5383620 10.2gb
2017.01.24-3 19    r  STARTED 5383770  5.7gb
2017.01.24-3 19    p  STARTED 5383770  5.5gb
2017.01.24-3 6     r  STARTED 5389052 10.2gb
2017.01.24-3 6     p  STARTED 5389052  5.8gb
2017.01.24-3 9     p  STARTED 5382316   10gb
2017.01.24-3 9     r  STARTED 5382316  5.6gb
2017.01.24-3 1     p  STARTED 5384850 10.3gb
2017.01.24-3 1     r  STARTED 5384850   10gb
2017.01.24-3 2     p  STARTED 5384110 10.3gb
2017.01.24-3 2     r  STARTED 5384110  5.4gb
2017.01.24-3 14    r  STARTED 5382046  5.4gb
2017.01.24-3 14    p  STARTED 5382046  5.8gb
2017.01.24-3 10    p  STARTED 5383007 10.3gb
2017.01.24-3 10    r  STARTED 5383007  7.7gb
2017.01.24-3 4     r  STARTED 5385397  5.4gb
2017.01.24-3 4     p  STARTED 5385397  5.4gb
2017.01.24-3 7     p  STARTED 5382244  5.5gb
2017.01.24-3 7     r  STARTED 5382244 10.2gb
2017.01.24-3 16    r  STARTED 5381232 10.2gb
2017.01.24-3 16    p  STARTED 5381232  5.5gb
2017.01.24-3 18    r  STARTED 5379792  6.3gb
2017.01.24-3 18    p  STARTED 5379792  5.4gb
2017.01.24-3 12    p  STARTED 5380705  5.4gb
2017.01.24-3 12    r  STARTED 5380705  6.7gb
2017.01.24-3 17    r  STARTED 5382108  5.5gb
2017.01.24-3 17    p  STARTED 5382108  5.8gb
2017.01.24-3 0     r  STARTED 5385285 10.2gb
2017.01.24-3 0     p  STARTED 5385285 10.1gb

I tested the incoming data in logstash by dumping data into a file. I got a file of 290 MB with 377822 lines in 30 seconds. So there is no issue from Kafka as at a given time I am receiving 35000 events per second in my 3 Logstash servers but my Elasticsearch is able to index maximum of 10000 events per second.

Can someone please help me with this issue?

2

As you are running on VMs I would recommend verifying that the VMs are not over-allocated on the hosts so that they actually have access to the resources you have provisioned. Memory ballooning can cause severe performance problems.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.