Autodiscovering logstash nodes in Kubernetes

[NerdSec (Nachiket)]

@Marius_Iversen

issue: Complete support for SCRAM-SHA-256/512 in Kafka output · Issue #16723 · elastic/beats · GitHub
PR: https://github.com/elastic/beats/pull/12867

Yes, if this is available in FileBeat, then yes, this would solve a large part of the problem. Will test this and get back to you! Thanks! A follow up question, any plans on adding support for FileBeat to delete the files after reading?

@Marius_Iversen

I never mind the architecture choice of Beat - Logstash - ES if there is a need, but in general it causes more bottlenecks and more pieces to debug, especially on things like bulks and how the ACK works, filebeat sends to logstash, logstash sends to ES, ES acks to Logstash, Logstash acks to filebeat (from my understanding, though not 100% sure).

I completely agree, and hence the need to add Kafka to act as a buffer and implementing a lambda for sharing this data across teams.

@Marius_Iversen

For the hostPath part I want to dig into that one a bit more as well, not to dodge the question, but I rather want to provide a long-term answer to your question, rather than trying to fix something because of limitations that might not exist anymore, or has a better workaround.

Thank you for taking the time to do so.