I am configuring an AWS-based ELK stack where I want to send my Filebeat output to AWS SQS and then from AWS SQS to an AWS ELB facing multiple Logstash instances. I cannot find any blog or doc on this subject. Has someone configured Filebeat to send log messages to SQS? If yes, can someone share their configs? Thank you in advance.
Currently Filebeat only supports Elasticsearch and Logstash as output. We are adding Kafka and Redis for the next major release. SQS is not supported.
I was going to pursue the same type of architecture, but I suppose I'll pursue Redis as a queueing service. Any plans to support SQS in upcoming releases?
Currently no plans to add SQS
Hi @Ruflin,
What's the way to submit an official enhancement request for adding SQS support to Beats?
We found Filebeat much more effective than Logstash, yet we are unable to switch and enjoy other Beats as well.
Thanks,
D.
You can open a feature request in the beats repo. But before you do, I recommend reading this entry here https://github.com/elastic/beats/issues/581#issuecomment-247034174 and the linked issue. I think it still holds true.
What I recommend you to do is having filebeat on all your edge nodes and sending data to a few LS nodes to the output it to SQS.