# Tuning Logstash for optimal throughput for ELK pipeline

**URL:** <https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007>\
**Category:** Logstash\
**Created:** [February 26, 2020, 10:33am UTC](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007 "2020-02-26T10:33:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kumarvikash1](https://avatars.discourse-cdn.com/v4/letter/k/839c29/32.png) [@kumarvikash1](https://discuss.elastic.co/u/kumarvikash1)\
**Post date:** [February 26, 2020, 10:33am UTC](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007/1 "2020-02-26T10:33:08Z")

</div>

We have Kafka-\>ELK pipeline and logstash is one of the key component in between Kafka and Elastic Search. Looks like Logstash is not able to process data at the same rate as injecting into kafka.  
We have below logstash config details:

input {  
kafka {  
bootstrap\_servers =\> "localhost:9092"  
consumer\_threads =\> 24  
topics =\> ["test0"]  
}  
}

filter {  
json {  
source =\> "message"  
target =\> "log"  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
}

Total CPU cores in localhost is 112.

Do we need to make any specific **INPUT** and **OUTPUT** config changes to improve Kafka-\>ELK pipeline throughput ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 26, 2020, 10:35am UTC](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007/2 "2020-02-26T10:35:55Z")

</div>

Logstash can only process at the speed Elasticsearch is able to consume data. How have you determined Elasticsearch is not the bottleneck?

---

<div class="post-metadata">

**Author:** ![kumarvikash1](https://avatars.discourse-cdn.com/v4/letter/k/839c29/32.png) [@kumarvikash1](https://discuss.elastic.co/u/kumarvikash1)\
**Post date:** [February 26, 2020, 11:34am UTC](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007/3 "2020-02-26T11:34:49Z")

</div>

Thanks for the quick reply Chris.  
I am new to ELK pipeline so please expect some basic queries ( although i have done some research ).

You are right, even ES can be bottlenecks, but CPU utilization for ES is ~10% ( 1 Cluster, 5 shards, ES 7.5.1 , pipeline.workers=112, pipeline.batchsize=125, concumer threads=48)

producer`script with 112 threads & 100000000 num-records are working fine at producer side but consumer server with Kafka-ELK components shows very less CPU utilization ( ~20%) with Logstash & ES as ~10% cpu utilization.

So my main concern is to find golden config for logstash and ES for optimal throughput for Kafka-ELK pipeline.

---

<div class="post-metadata">

**Author:** ![kumarvikash1](https://avatars.discourse-cdn.com/v4/letter/k/839c29/32.png) [@kumarvikash1](https://discuss.elastic.co/u/kumarvikash1)\
**Post date:** [February 28, 2020, 8:21am UTC](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007/4 "2020-02-28T08:21:55Z")

</div>

Hi Chris,

Did i confuse you with my question ? Do you need any other details ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2020, 8:22am UTC](https://discuss.elastic.co/t/tuning-logstash-for-optimal-throughput-for-elk-pipeline/221007/5 "2020-03-27T08:22:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.