# Logstash decrease write rate when rabbitmq full

**URL:** <https://discuss.elastic.co/t/logstash-decrease-write-rate-when-rabbitmq-full/85064>\
**Category:** Logstash\
**Created:** [May 9, 2017, 10:47am UTC](https://discuss.elastic.co/t/logstash-decrease-write-rate-when-rabbitmq-full/85064 "2017-05-09T10:47:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![guy\_es](https://avatars.discourse-cdn.com/v4/letter/g/67e7ee/32.png) [@guy\_es](https://discuss.elastic.co/u/guy_es)\
**Post date:** [May 9, 2017, 10:47am UTC](https://discuss.elastic.co/t/logstash-decrease-write-rate-when-rabbitmq-full/85064/1 "2017-05-09T10:47:30Z")

</div>

Hi all,

I use logstash 2.4 to forward docs from RabbitMQ to Elasticsearch 2.4, I see correlation between how much the queue is full to logstash write rate. When the queue gets full the write rate significantly decrease and vice versa, did someone here encounter this issue or have a suggestion why might it happen?

logstash servers are idle, ES looks OK ...

input {  
rabbitmq {  
host =\> "host\_name"  
queue =\> "queue\_name"  
exchange =\> "exchange\_name"  
codec =\> "json"  
durable =\> true  
prefetch\_count =\> 1000  
threads =\> 8  
}  
}

output {  
elasticsearch {  
hosts =\> ["elasticsearch24-hostname"]  
index =\> "events-%{+YYYY-MM-dd}-v%{schema\_version}"  
routing =\> "%{org\_pk}"  
document\_id =\> "%{doc\_id}"  
flush\_size =\> 1000  
idle\_flush\_time =\> 2  
document\_type =\> "c\_levent"  
}  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 14, 2017, 9:31am UTC](https://discuss.elastic.co/t/logstash-decrease-write-rate-when-rabbitmq-full/85064/2 "2017-05-14T09:31:41Z")

</div>

I would expect the indexing speed of Elasticsearch to determine the throughput here. If Elasticsearch is not able to keep up the queue in RabbitMQ will grow. As you are assigning document IDs at the application side, it is possible that the indexing throughput in Elasticsearch will deteriorate as the size of the index being written to increases, but this depends on how how large the indices grow to and how the ID is created. Are you seeing any correlation between the indexing throughput and point in time when a new time-based index is created? Do you have monitoring installed?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2017, 9:37am UTC](https://discuss.elastic.co/t/logstash-decrease-write-rate-when-rabbitmq-full/85064/3 "2017-06-11T09:37:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.