Preamble that I'm relatively new to Elasticsearch, load testing, and by no means a cloud infrastructure expert...
I've had success with esrally and creating a new track to test a custom search query load, both against a local node that rally creates and our "dev" cluster.
My next step is to try out distributed load drivers, with the ultimate goal of simulating current average load, then 10x, 100x, etc. and see where things fall apart.
I see that there is a documented limitation in using the docker image Running Rally with Docker - Rally 2.8.0.dev0 documentation and distributing the load test driver. It seems to me like running it in docker is such an obviously convenient way for temporarily standing up a distributed load test driver. I have some questions to make sure I'm not missing something...
Is there any way to work around this limitation? In other words, could I package my own docker image with the needed dependencies whose entrypoint was the appropriate
esrallydcommands? Why wouldn't that work?
The examples and other posts I find about distributing the load driver are typically setting up 2-3 instances. Since my goal is to simulate production traffic, I think many more instances would be appropriate. The rate of requests per client would obviously be much less, but I'm sure there are many ES clusters that have thousands of clients making requests per second. A distributed load driver network of 100 clients seems feasible. What is an appropriate way to size the distributed load driver network?
If there are indeed rally tests using tens or hundreds of distributed load drivers, what is recommended to set it up? A container just seems like a natural part of automating that.
If we have an ingress/load balancer already configured via cloud infrastructure, such that our clients access via a single address, is there any point in running with the individual hosts separated out in the
Sorry for the long post and thanks for any help!