I am missing a setting or two on logstash I am sure but don't know which. My elastic cluster typically indexes at about 12000 docs/sec. I have Logstash pulling from kafka and sensing to Elastic. Seeing speeds of 250/sec.
I have 8 partitions in Kafka
My pipeline.workers: 8
My pipeline.batch.size: 5000
jvm has 16GB of Ram. I am using SSDs, there is no way its this slow. Something is happening.
What am I missing? Thanks in advance!
If I run logstash as a service its very slow. If I run the same config from command like its fast. Here is a graph that shows the difference. Can someone tell me why?
Does your OS run services at a lower priority than interactive shells? If so, it may be seeing CPU contention.
It should not. How do I check this? I am running Centos 7.5 with default configs. Infact BARELY any CPU was used when its running as service and plenty CPU is available.
I would read the man pages for which service manager you use.
Its not running at a lower priority, when I run top it shows the java process running just fine, I just dont see a high enough CPU usage which means it could be defaulting to some other config? I dont know.
Anyone?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
