# Kafka optimal configuration **URL:** \ **Category:** Logstash\ **Created:** [November 13, 2015, 7:37pm UTC](https://discuss.elastic.co/t/kafka-optimal-configuration/34554 "2015-11-13T19:37:08Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [November 13, 2015, 7:37pm UTC](https://discuss.elastic.co/t/kafka-optimal-configuration/34554/1 "2015-11-13T19:37:08Z")
I am trying to use Kafka to replace redis. I am curious if anyone can help with this question I have three brokers that would have 4 topics. Each topic would have dedicated three logstash indexers. My question should the topics have 3 partitions and each indexer goes to 1 partition with 1 consumer\_thread? or should expand it a little bit. IE topic has 6 partitions (2 per broker) and each indexer has 2 consumer\_threads? i know the total # of consumer\_threads should equal total partitions of a topic but dont know what scenario leads to higher throughput. Any guidance would be great ---
**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\ **Post date:** [November 19, 2015, 7:00pm UTC](https://discuss.elastic.co/t/kafka-optimal-configuration/34554/2 "2015-11-19T19:00:45Z")
Your primary bottleneck is likely going to be Elasticsearch. Also, it would help if you let us know 1) cloud or data center and 2) cpu and memory sizes. I am assuming you are in a data center. With that said, lets consider the questions. First, always leave the logstash-input-kafka consumer\_threads at 1. That option should be removed and I keep saying it here but don't do a PR, so apologies for that. The short explanation here is that Kafka, in the background, creates a consumer thread for each partition it has assigned. Each consumer thread pushes its messages in the internal buffer into Logstash's processing chain (input\>filter\>output). Because each consumer thread has it's own collection of threads there is no need to add the additional "consumer\_threads" as threading is already happening. That aside, the scenarios we are looking at are three partitions with three consumers or six partitions with three consumers. The primary reason to have more partitions than the maximum number of consumers you are running is to allow more throughput later if you added more consumers. For example if you ran six logstash servers each with a logstash-input-kafka consumer instead of just three. Raising the number of partitions beyond three when you are only running with three consumers will likely not give you much of a benefit from the consumption perspective, in fact it'll probably slow things down as there will be parallel context switches going on in the background. Also consider that Kafka is just streaming offsets from the brokers so breaking up partitions can causes more disk IO thrashing if the events are not in cache. Please keep in mind that "slowing down" is in PC speeds, not human so the impact could be negligible. Also if your Kafka boxes are running SSDs and a ton of memory, more partitions isn't going to hurt much and can give you headroom for the future. Consider scanarios where many systems are producing messages for Kafka, having more partitions can spread the load across the brokers. One scenario where you may want to have more partitions is if your logstash server has multiple CPU cores. You would probably want 1 to 2 cores for logstash and then match the rest for kafka. IE if each box is a 4 core, you may want to consider a setup of 2x partitions per logstash instance. So six partitions with three four core machines. you would have two cores handling all the kafka threads and two filtering and talking to Elastchsearch. If you are on the cloud, I generally suggest using 1 CPU machines and then set your logstash autoscaling group to scale on CPU consumption with a minimum of 1 and a max equal to the number of partitions. Then your setup scales on load. When peak is hitting more instances will fire, etc. Hope this helps! Sincerely, Joe Lawson ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:22am UTC](https://discuss.elastic.co/t/kafka-optimal-configuration/34554/3 "2017-07-06T05:22:08Z")