Hi,
I am new to ELK and I am running ELK in a CentOS 7 environment. I have 4 CPU cores in this machine and with the default configuration of Logstash, the CPU usage is always very high.
Knowing that running Logstash is I/O intensive and multiline consumes a lot of CPU resource, here below is my Logstash configuration file. May I know how to optimize the CPU utilization for Logstash?
input {
file {
path => ["java.log"]
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
patterns_dir => "/etc/logstash/custom-patterns/custom"
pattern => "^%{NUMBER20} "
negate => true
what => previous
}
type => "xservice"
}
}filter {
if [type] == "xservice" {
mutate {
replace => {document_type => "xservice"}
gsub => ["message", "\n", ""]
}grok {
patterns_dir => "/etc/logstash/custom-patterns/custom"
match => {"message" => "%{JAVALOG}"}
}date {
match => ["timestamp","YYYY-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
remove_field => ["timestamp"]
}
}
}output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{document_type}-%{+YYYY.MM.dd}"
}
}
Also, it was previously very slow in log ingestion until I made some changes to the pipeline.batch.size, pipeline.batch.delay and JVM heap size, it got better in performance but I am worried with the CPU utilization.
Please help. Thank you.