We are looking into deploying APM throughout our fleet of agents (+300), but we are discovering some challenges.
Context: These agents are running on appliances we have deployed at our customers. We are looking into using Synthetic Monitors to follow up on uptime and other metrics from our applications running on these appliances.
The issue:
We have a pretty logical grouping of agents and fleet policies: we have our "main" production policy which contains all customer appliances, and we have some specific "internal" and "development" policies for our internal infrastructure. Worked pretty well. However, in the Synthetic Monitoring documentation I found that for monitoring and alerts to work, these monitors (Private Locations) need to be linked with agent policies having only one single agent.
This is problematic since this would raise complexity of agent deployment dramatically if we want to deploy APM monitors to each appliance.
Is this a hard requirement? Could we go around this with custom alerting rules etc.? Any feedback on this topic is most welcome.