I've got an issue with Logstash 5.x on an RHEL7 VM (8 vCPU, 2GB memory) on vSphere 6 hosts with E5-2698 v4 Xeon CPUs.
I've installed the filter plugin and the Netflow plugin - basic gist is that I'm ingesting Netflow and using the filter plugin to enrich the Netflow with one additional bit of information before sending it to Elasticsearch 5.x (instead of just having a number for IPPROTOCOL, putting in a name as well).
I've got one firewall sending Netflow to it (data rate is about 6.5Mbit worth of Netflow traffic), and despite my best efforts Logstash is continually dropping UDP packets and CPU usage is incredibly high. No settings adjustments that I've made have managed to make things much better. Below are some configuration snippets and show outputs. I should note that Elasticsearch is practically idle, with maybe 23% CPU usage on a single vCPU (machine has 8x vCPU).
Any assistance is greatly appreciated.
netstat -suna
Udp:
267034 packets received
147276 packets to unknown port received.
254987 packet receive errors
130 packets sent
0 receive buffer errors
0 send buffer errors
netstat -neopa | grep udp
udp 202368 0 0.0.0.0:2055 0.0.0.0:* 995 35399 2813/java off (0.00/0/0)
curl -XGET http://localhost:9600/_node/hot_threads?pretty=true&threads=4
[1] 3144
{
"host" : "netflow1.blah.com",
"version" : "5.5.1",
"http_address" : "127.0.0.1:9600",
"id" : "1fc0627e-1a59-4e72-9bde-2e7aed695a03",
"name" : "netflow1.blah.com",
"hot_threads" : {
"time" : "2017-07-27T13:54:50-05:00",
"busiest_threads" : 3,
"threads" : [ {
"name" : "Ruby-0-Thread-3",
"percent_of_cpu_time" : 0.01,
"state" : "timed_waiting",
"path" : "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:22",
"traces" : [ "java.lang.Object.wait(Native Method)", "org.jruby.RubyThread.sleep(RubyThread.java:1002)", "org.jruby.RubyKernel.sleep(RubyKernel.java:803)" ]
}, {
"name" : "<udp.0",
"percent_of_cpu_time" : 71.66,
"state" : "runnable",
"traces" : [ "java.lang.Throwable.getStackTraceElement(Native Method)", "java.lang.Throwable.getOurStackTrace(Throwable.java:827)", "java.lang.Throwable.getStackTrace(Throwable.java:816)" ]
}, {
"name" : "Ruby-0-Thread-25",
"percent_of_cpu_time" : 0.0,
"state" : "runnable",
"path" : "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/reactor.rb:136",
"traces" : [ "sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)", "sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)" ]
} ]
}
}
cat /etc/logstash/jvm.options
-Xms1024m
-Xmx1536m
cat /etc/logstash/conf.d/output.conf
output {
elasticsearch {
hosts => ["elasticsearch.blah.com"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}
cat /etc/logstash/logstash.yml
pipeline.batch.size: 50000