Using 3-node logstash to consume kafka data, logstash cpu usage is very high, and there are a large number of thread GC
logstash configuration: 3 node , 16C 32G
jvm -Xms16g -Xmx16g
Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them
What do the Logstash logs show?
logstash shows normal, no error
Up to a certain point (and I don't think 16 cores is it), a parallel garbage collector can use one thread per core, so having 16 GC threads looks normal to me. Having the GC threads use 3.7% of the CPU seems high. You would have to review the GC logs to see what is happening (which I used to do for a living, but that was years ago, with collectors that are not much used now).
What does your pipelines configuration looks like?
I added consumer_threads in logstash input kafka，but now output Elasticsearch has many rejected
This is my pipelines
- pipeline.id: case_logs
- pipeline.id: other_logs
You need to share the content of the
.conf files, if your logstash is using a lot of cpu the issue could be in one of your filters for example.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.