Hello! I've been setting up a Elastic Stack over the last few months to monitor infrastructure (using MetricBeat + Logstash on my clients, sending that data to Elasticsearch + Kibana).
I'm interested in expanding the monitoring capability to include APM on our Django applications and Python workers.
When looking at the API set up for Django, it seems like a custom APM middleware will send data to an APM server, which in turn will process that data and store it in Elasticsearch. I'm wondering, what is the architectural reason for having this dedicated APM Server process? My team and I are trying to reduce the footprint of our monitoring infrastructure, and we are very weary about adding an extra process to the stack.
The way I understand MetricBeat and Logstash to work is that these applications run on the client machine and send data directly to Elasticsearch; then using Kibana dashboards I can view the data.
My question is, what is the added value that the dedicated APM server brings? Why couldn't we just send the APM data directly from the client to Elasticsearch like we do for Logstash and Metricbeat? (I assume this was a well-thought out decision and I'm just looking to the rationale so I can relay this to the team!)