Rsyslog + logstash issue -rsyslog stalls when logstash -elasticsearch communication breaks

[ajayraghuraj]

Hello,

Here is a rsyslog-logstash integration scenario. rsyslog is pumping filtered messages to logstash . logstash in turn sends data to elasticsearch. Something like a storage full event occurs on the elasticsearch host. This is when it seems like elasticsearch blocks further updates from logstash until the underlying storage problem is solved. Eventually logstash stalls rsyslog . rsyslog stops from logging messages .

what could be the issue here ?

One theory is that Logstash puts back pressure on rsyslog to stall data flowing into Logstash.

could this be the reason why rsyslog hangs ?

here is a snippet of the errors from logstash

[INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

Thanks

[magnusbaeck (Magnus Bäck)]

could this be the reason why rsyslog hangs ?

Yes. ES applies backpressure to Logstash which applies backpressure to rsyslog.

[ajayraghuraj]

thank you for the reply. we introduced action queues in rsyslog to overcome this problem. let us see how it works . i will keep the forum posted . here are a couple of links for reference

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/s1-working_with_queues_in_rsyslog

RSyslog Documentation

has anyone implemented this before with logstash ?