[heartbeat]how to get performace test on heartbeat

Hi Elastic friends,
Have a question for the performance of heartbeat.
Assuming there’re 90K targets need to be monitored via heartbeat, in some case, like there’re 30% to 50% targets, maybe even worse, there’re 60%~80% targets are very slow to access from heartbeat. what will happen to heartbeat? And is there any easy way to simulate this scenario?
We want to reproduce this scenario without losing any workload when targets count is nearly 90K.

Any suggestions?

Hello Guys, any thoughts here?

Apologies for the delay. Assuming HTTP or TCP, If some hosts are slower than others the main consequence of that will be heartbeat will will keep additional file descriptors allocated for each host. This will require some level of benchmarking on your end if you want to be confident that it will work fine.

t's a tough thing to simulate but it is possible. What I would do is spin up a few VM's on the cloud as test targets and send 20,000 monitors to each. I would just run nginx on the VM's. Then, I would generate a simple heartbeat config with the full list of monitors. To simulate a slow connection you could use qdiscs with linux. For an example see: linux - Simulating a slow connection with tc - Server Fault .

You may also want to set heartbeat.scheduler.limit (see docs).

ICMP does not require the allocation of file descriptors so it scales much further.

We'd be really curious to see the results of a test like this!

Hi Andrew, thanks for your suggestion.

However, I was wondering if we could add an option from heartbeat side, like delay as the same level as timeout ?
That means HB has to wait for the delay period then it starts the check for every single icmp/port/url check.
Then we can set the value of delay very close to timeout , once we do that for the targets (like 30%, 40% of all targets) we want to test on, in this case, heartbeat will unable to get the response for them in time.

Appreciate if you could comment on this solution.

Hello Andrew, very appreciated if you could comment on our proposal.

Apologies for the delay, I'm just back from vacation :-).

I'm unclear as to what the differences between delay and the current schedule option.