This is not a question about a specific setup, it's more or less me trying to understand the intended use of Beats / Fleet in distributed setup.
In the past I built several setups where I used Logstash to transport log data from remote sites to a central Elasticsearch cluster. By "remote sites" I mean everything from "cloud to on-prem", remote sites via VPN or just a DMZ that's heavily guarded by firewalls.
I wonder how I could manage that when I replace Logstash with Beats / Elastic Agent and Ingest Pipelines.
I know, I can have a beat write to Kafka which would be a way of transporting data through any kind barrier but how can I read from Kafka without Logstash? Is there a beat I missed or some functionality in Elasticsearch?
Same goes for sending data with beats from heavily guarded internal networks. I often see hosts that are not allowed to connect to the outside world. How could I get beats send data to e.g. Elastic Cloud when I'm not allowed to connect directly? Is there any way I can use proxies that I haven't found?
Same goes for fleet. How can I use it to configure agents that are not allowed to connect to Elasticsearch / Kibana? Is there another proxy mechanism I missed?
Please don't get me wrong. I love Logstash and you couldn't make me happier than by telling me I should just use Logstash. The reason why I'm asking is that I see less and less Logstash love in Elastic Agent and Fleet and I don't want to miss out all the new possibilities just because I want to keep my beloved Logstash.
Sorry if I asked something obvious.