Hello,
Im kinda new to the elastic suite, I’ve been doing some research on my own trying to solve the issue but didn’t have any luck, so im kindly asking anyone here to help me out in this matter.
Scenario:
Simple: Two servers connected back to back:
ServerA -10g- ServerB
ServerA running Elasticsearch
[root@elasticsearch elasticsearch]# /usr/share/elasticsearch/bin/elasticsearch --version
Version: 2.2.2, Build: fcc01dd/2016-03-29T08:49:35Z, JVM: 1.8.0_65
ServerB running Logstash
[root@logstash conf.d]# /opt/logstash/bin/logstash --version
logstash 2.2.4
Purpose
Elasticsearch search should be able to index around a minimum of 1500 documents per second.
Issue
On ServerB I have Logstash and in ServerA I have Elasticsearch.
After executing a Perl script that generates around 4.5 million logs (it sends 1000 logs every second) in around 78 minutes I noticed that after the script is finished, less than 2.8m documents are indexed, meaning that a little les than 2M documents get lost. There is a bottleneck somewhere.
I discarded logstash as the bottlenext, the reason is because I was able to redirect Logstash output to a document and was able to successfully write 1M, 2M, 8M lines into the document without a single loss.
Logstash configuration
input {
udp {
port => 5002
}
}
filter {
grok {
match => { "message" => "%{NUMBER:Log} %{WORD:MAC} %{NUMBER:Var0} %{NUMBER:Var1} %{NUMBER:Var2} %{NUMBER:Var3} %{NUMBER:Var4} %{NUMBER:Var5} %{NUMBER:Var6} %{NUMBER:Var7} %{NUMBER:Var8} % {NUMBER:Var9}" }
}
}
output {
elasticsearch { hosts => ["10.10.40.225:9200"] }
}
Elasticsearch configuration
Elasticsearch configuration is pretty much default except for the following tweaks I’ve done based on some reading I’ve been doing about Elasticsearch performance:
-File Descriptors, increased to 64k
[root@elasticsearch ~]# cat /proc/18045/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 30504 30504 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 30504 30504 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
[root@elasticsearch ~]#
-HEAP, increased to 4g (half of current RAM on the server)
[root@elasticsearch ~]# !ps
ps -ef | grep -i elasticse
avahi 799 1 0 2016 ? 00:00:01 avahi-daemon: running [elasticsearch.local]
elastic+ 18045 1 99 19:25 ? 00:00:04 /bin/java -Xms4g -Xmx4g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.2.2.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch