# Tweaking pipeline performance

**URL:** <https://discuss.elastic.co/t/tweaking-pipeline-performance/308801>\
**Category:** Logstash\
**Created:** [July 4, 2022, 9:30am UTC](https://discuss.elastic.co/t/tweaking-pipeline-performance/308801 "2022-07-04T09:30:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [July 4, 2022, 9:30am UTC](https://discuss.elastic.co/t/tweaking-pipeline-performance/308801/1 "2022-07-04T09:30:04Z")

</div>

Hello,

I'm trying to get the hang off tweaking logstashs performance. I searched the net but unfortunately, I can't seem to find a good guide on how to tweak and actually measure the performance of a pipeline.  
I can't use the pipeline management stuff inside Kibana which is why I'm doing with a curl pipelines command.

Currently, my problem is the following:

I got a dedicated redis and a dedicated logstash cluster. The logstash cluster pulls the data from the redis cluster, does it's thing and then writes to a dedicated elasticsearch cluster.  
Even if the redis cluster has many keys "building up" or waiting to get pulled, the CPU utilization of the logstash cluster is only at ~ 50%.  
One thing I found is by diving the queue\_push\_duration\_in\_millis through duration\_in\_millis, I should be able to see if Logstash is waiting for workers. I sometimes get a value of 4,X which would mean logstash is waiting a lot of its time, correct?

I'm unsure how I could tweak the settings and would really love a useful guide which I could use.

Thanks in advance  
Ossenfeld

---

<div class="post-metadata">

**Author:** ![anon90868141](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon90868141](https://discuss.elastic.co/u/anon90868141)\
**Post date:** [July 6, 2022, 4:48am UTC](https://discuss.elastic.co/t/tweaking-pipeline-performance/308801/2 "2022-07-06T04:48:52Z")

</div>

In on of my pipelines, I get the following results:

```
"distributor_lan_filebeat" : {
  "events" : {
    "duration_in_millis" : 174876784,
    "in" : 5250849,
    "queue_push_duration_in_millis" : 831997131,
    "filtered" : 5226753,
    "out" : 5226753

```

These seem very bad, but where would I start tweaking?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 3, 2022, 4:49am UTC](https://discuss.elastic.co/t/tweaking-pipeline-performance/308801/3 "2022-08-03T04:49:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.