Heartbeat Maximum No. of Monitors

Elastic Stack Beats
1

Saw this thread way back 4 years ago but it seems like there's no update or similar thread regarding this.

We wanted to monitor millions of websites/services atleast once a day.
We can run a job that can configure the heartbeat yml but, is it possible for heartbeat to monitor 2 millions of monitors?

2

Apologies for the delay. Heartbeat can monitor millions of monitors however you will need more than a single heartbeat instance. If you split the monitors up across multiple heartbeat instances the main limiting factor will be how big of an elasticsearch cluster you have to track all this data. You may even want to have more than one elasticsearch cluster and use cross cluster search depending on your use case. Can you tell us more about your business use case? That would help us understand what kind of architecture would make the most sense.

Additionally, the uptime UI in kibana is not meant to handle use cases of this scale and will probably not work. However, you could create a dashboard in kibana that would be able to provide some level of metrics for this data.

One thing that will make a big difference is whether you are monitoring ICMP, TCP, or HTTP.

ICMP is very lightweight to monitor.

TCP and HTTP are limited by the number of file descriptors your system can use simultaneously. You will certainly have to adjust OS limits to their max to handle this case if you are using TCP and HTTP.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.