Hi
I am setting up filebeat -> logstash -> rabbitMQ ->elasticsearch
some how logstash simple config is using 100%+ cpu
is this normal ? or am I missing some thing
filebeat ( V1.3.1)
logstash ( V2.4.1)
cpu : AMD Opteron(tm) Processor 6376 32core
ram: 128G
logstash config ---------
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/cert/cert.crt"
ssl_key => "/cert/cert.key"
ssl_key_passphrase => "passphrase"
}
} #end of input
filter {
} #end of Filter
output {
}#end of output
LS_HEAP_SIZE=4G
ps -ef|grep logstash
conse 16587 1 99 12:27 pts/0 00:28:01 /app/jdk1.8.0_51/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Xmx4G -Xss2048k -Djffi.boot.library.path=/app/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/logstash/heapdump.hprof -Xbootclasspath/a:/app/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/app/logstash/vendor/jruby -Djruby.lib=/app/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /app/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /app/logstash/config/aa.conf --verbose -l /data/logstash/logstash.log
and most of cpu are used by beat input
top -Hp 16587
26473 conse 20 0 11.729g 1.017g 20488 S 70.1 0.8 3:10.07 [main]<beats
16632 conse 20 0 11.729g 1.017g 20488 S 4.0 0.8 1:02.98 java
16631 conse 20 0 11.729g 1.017g 20488 S 2.0 0.8 0:19.29 java
16611 conse 20 0 11.729g 1.017g 20488 S 1.7 0.8 0:11.49 java
16613 conse 20 0 11.729g 1.017g 20488 S 1.7 0.8 0:11.74 java
filebeat config --------
############################# Shipper #########################################
shipper:
name: "172.16.100.203"
tags: ["epsLog"]
############################# Filebeat
filebeat:
spool_size, idle_timeout is for network
spool_size: 4089
idle_timeout: 5s
registry_file: /data/beat/filebeat1.point
prospectors:
-
paths:
- /manager/log/*
exclude_lines: ["^#"]
exclude_files: [ "dnscount.log", "pe.log", "stdout.log", "communication.log", "dns-passivedns.log", "pcr.log", "stderr.log" ]
fields_under_root: true
backoff: 1s
scan_frequency: 500ms
harvester_buffer_size: 6000
tail_files: true
############################# Output
output:
Logstash as output
logstash:
# The Logstash hosts
hosts: [ "5.5.5.5:5044" ]
# Number of workers per Logstash host.
worker: 2
# Optional load balance the events between the Logstash hosts
loadbalance: false
# Optional index name. The default index name depends on the each beat.
# For Packetbeat, the default is set to packetbeat, for Topbeat
# top topbeat and for Filebeat to filebeat.
index: logs
compression_level: 0
bulk_max_size: 2048
tls:
# List of root certificates for HTTPS server verifications
certificate_authorities: ["/cert/cert.crt"]
insecure: true
min_version: '1.2'
file:
path: "/data/beat"
filename: eps.file
rotate_every_kb: 50000 #=50M
number_of_files: 7
############################# Logging
There are three options for the log ouput: syslog, file, stderr.
Under Windos systems, the log files are per default sent to the file output,
under all other system per default to syslog.
logging:
to_syslog: false
to_files: true
files:
path: /data/beat
name: eps.log
rotateeverybytes: 50485760 # = 50MB
keepfiles: 7
level: info
from other server filebeat is sending about 5000 +- event per second
2017-02-02T12:42:50+09:00 INFO Events sent: 4089
2017-02-02T12:42:50+09:00 INFO Registry file updated. 60 states written.
2017-02-02T12:42:51+09:00 INFO Events sent: 4089
2017-02-02T12:42:51+09:00 INFO Registry file updated. 60 states written.
2017-02-02T12:42:51+09:00 INFO Events sent: 4089
2017-02-02T12:42:51+09:00 INFO Registry file updated. 60 states written.
2017-02-02T12:42:52+09:00 INFO Events sent: 4089
2017-02-02T12:42:52+09:00 INFO Registry file updated. 60 states written.
2017-02-02T12:42:53+09:00 INFO Events sent: 4089
2017-02-02T12:42:53+09:00 INFO Registry file updated. 60 states written.
2017-02-02T12:42:53+09:00 INFO Events sent: 4089
2017-02-02T12:42:53+09:00 INFO Registry file updated. 60 states written.