Logstash slow

I am missing a setting or two on logstash I am sure but don't know which. My elastic cluster typically indexes at about 12000 docs/sec. I have Logstash pulling from kafka and sensing to Elastic. Seeing speeds of 250/sec.

I have 8 partitions in Kafka
My pipeline.workers: 8
My pipeline.batch.size: 5000

jvm has 16GB of Ram. I am using SSDs, there is no way its this slow. Something is happening.

What am I missing? Thanks in advance!

If I run logstash as a service its very slow. If I run the same config from command like its fast. Here is a graph that shows the difference. Can someone tell me why?

Does your OS run services at a lower priority than interactive shells? If so, it may be seeing CPU contention.

It should not. How do I check this? I am running Centos 7.5 with default configs. Infact BARELY any CPU was used when its running as service and plenty CPU is available.

I would read the man pages for which service manager you use.

Its not running at a lower priority, when I run top it shows the java process running just fine, I just dont see a high enough CPU usage which means it could be defaulting to some other config? I dont know.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.