Good day, dear forum users!
My logs transfer scheme is as follows:
App servers -> AWS SQS (queue) -> Logstash (input => sqs, output => elasticsearch) -> AWS Elasticsearch
Sometimes Elasticsearch is blocked for writing (for example, due to lack of space) and here I have questions.
- What happens in Logstash? Do messages get in the in-memory Logstash queue?
- Is there a limit to these messages (RAM size/etc)?
- Why with a blocked Elasticsearch writing, I see that the SQS queue is growing, and Logstash does not fail?
- If it does not fall, then maybe the amount of memory allocated for the JVM heap is filled and then Logstash stops receiving messages?
Help find out the truth. Thank you in advance for your answers.