I am missing a setting or two on logstash I am sure but don't know which. My elastic cluster typically indexes at about 12000 docs/sec. I have Logstash pulling from kafka and sensing to Elastic. Seeing speeds of 250/sec.
I have 8 partitions in Kafka
My pipeline.workers: 8
My pipeline.batch.size: 5000
jvm has 16GB of Ram. I am using SSDs, there is no way its this slow. Something is happening.
What am I missing? Thanks in advance!