# Rsyslog + logstash issue -rsyslog stalls when logstash -elasticsearch communication breaks

**URL:** <https://discuss.elastic.co/t/rsyslog-logstash-issue-rsyslog-stalls-when-logstash-elasticsearch-communication-breaks/148965>\
**Category:** Logstash\
**Created:** [September 18, 2018, 11:09am UTC](https://discuss.elastic.co/t/rsyslog-logstash-issue-rsyslog-stalls-when-logstash-elasticsearch-communication-breaks/148965 "2018-09-18T11:09:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajayraghuraj](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@ajayraghuraj](https://discuss.elastic.co/u/ajayraghuraj)\
**Post date:** [September 18, 2018, 11:09am UTC](https://discuss.elastic.co/t/rsyslog-logstash-issue-rsyslog-stalls-when-logstash-elasticsearch-communication-breaks/148965/1 "2018-09-18T11:09:33Z")

</div>

Hello,

Here is a rsyslog-logstash integration scenario. rsyslog is pumping filtered messages to logstash . logstash in turn sends data to elasticsearch. Something like a storage full event occurs on the elasticsearch host. This is when it seems like elasticsearch blocks further updates from logstash until the underlying storage problem is solved. Eventually logstash stalls rsyslog . rsyslog stops from logging messages .

what could be the issue here ?

One theory is that Logstash puts back pressure on rsyslog to stall data flowing into Logstash.

could this be the reason why rsyslog hangs ?

here is a snippet of the errors from logstash

[INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})  
[INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})  
[INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>1}  
[INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>1}  
[INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>1}  
[INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 20, 2018, 5:33pm UTC](https://discuss.elastic.co/t/rsyslog-logstash-issue-rsyslog-stalls-when-logstash-elasticsearch-communication-breaks/148965/2 "2018-09-20T17:33:06Z")

</div>

> could this be the reason why rsyslog hangs ?

Yes. ES applies backpressure to Logstash which applies backpressure to rsyslog.

---

<div class="post-metadata">

**Author:** ![ajayraghuraj](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@ajayraghuraj](https://discuss.elastic.co/u/ajayraghuraj)\
**Post date:** [September 22, 2018, 6:35am UTC](https://discuss.elastic.co/t/rsyslog-logstash-issue-rsyslog-stalls-when-logstash-elasticsearch-communication-breaks/148965/3 "2018-09-22T06:35:19Z")

</div>

thank you for the reply. we introduced action queues in rsyslog to overcome this problem. let us see how it works . i will keep the forum posted . here are a couple of links for reference

[https://access.redhat.com/documentation/en-us/red\_hat\_enterprise\_linux/7/html/system\_administrators\_guide/s1-working\_with\_queues\_in\_rsyslog](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/s1-working_with_queues_in_rsyslog)

> **[RSyslog Documentation](https://www.rsyslog.com/doc/v5-stable/concepts/queues.html)**
>
> The rocket-fast system for log processing

has anyone implemented this before with logstash ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 20, 2018, 6:35am UTC](https://discuss.elastic.co/t/rsyslog-logstash-issue-rsyslog-stalls-when-logstash-elasticsearch-communication-breaks/148965/4 "2018-10-20T06:35:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.