We've set up an Elastic APM cluster composed of three APM servers fronted by Haproxy.
Load testing result has been great on this stack, and I could see that all APM Server instances are happily distributing the load against a cluster of seven Elasticsearch nodes (3 master only, 4 data only). I confirm this in the Haproxy response headers.
One thing I noticed is that Kibana monitoring is only showing a single APM Server instance, which I suspect is the first APM server to register in Elasticsearch. How do i get the other APM Server instances to display in Kibana? APM Servers count is "1"
Our set up:
Haproxy ----> (3) APM Servers ----> (7) Elasticsearch <---Kibana
All are installed via docker.
I've tested two configuration for the APM Servers: via individually started containers and as a docker swarm scale up. Results are still the same, only one APM Server instance is displayed in the Kibana monitoring page (using Docker container ID). APM server containers are all built on the same image, all share the same configuration except for the individually started containers which have their own published IP address being used by Haproxy.
Kibana correctly shows 7 Elasticsearch server instances.
Forgot to mention that Machine Learning shows three beat names corresponding to the docker ids of the APM Server containers.
Any help is appreciated as we'd like to be able to monitor all APM Server instances in Kibana.