Rally and Httperf

[jlwysf (Eric)]

I'm new to rally, I used to use httperf, I have some questions hope I can get an answer here.

1. In httperf we can set concurrent connections, does Rally also support that?

   • If yes,How many concurrent Connection by default for each operation and where can I set it?

2. I do see we can set up clients in schedule, does one client means one node in elasticsearch cluster?

3. I have two existing ES cluster and I would like test performances between these two clusters, I assume "pipeline-benchmark only " is the right option, however is there anyway Rally can help get CPU usage, Disk IO during testing?

Does these two testings can run in parallel? Because I always get race tun error;

Thanks

Eric

[dliappis (Dimitrios Liappis)]

Hello @jlwysf,

Some replies below (caveat: I am also currently learning Rally):

@jlwysf

I do see we can set up clients in schedule, does one client means one node in elasticsearch cluster?

No, client is an Elasticsearch client (they are actually implemented using the elasticsearch-python client).
When you increase the clients, e.g. for a bulk operation, you will see more python3 processes getting spawned.

The following links in docs and discuss replies (1, 2, 3) have more detail.

@jlwysf

In httperf we can set concurrent connections, does Rally also support that?

You should take a look at adjusting clients, as mentioned earlier and running tasks in parallel.

The documentation provides some examples and the amount of clients can be decided by Rally or be specified.

@jlwysf

is there anyway Rally can help get CPU usage, Disk IO during testing?

Rally has internal telemetry devices through which it stores a number of very useful metric keys, documented here. Among these metrics, you will a find a number of cpu utilization and disk_io related ones. See also this discuss reply.

Dimitris

[jlwysf (Eric)]

Hi Dimitris,

Thanks for replying me. I still confused about my second and last question and I haven't saw an answer,

• Regarding my second question, if "client" is the concurrent session you referred to, how many thread are related in one client, how can I know? Does one client is one process? If we only have one client , is that mean we only execute one concurrent session?

Regarding my last question,

• According to you answer, I haven't seen any index called "rally-*/metrics/" is that because my existing cluster is not provisioned by esrally or they are reside in somewhere else?

• According to this answer I have another question regarding how can I know how many indexing thread I have, does indexing thread is python es client you are talking about earlier?

• Basically I try to run esrally against two existing cluster separately, but I only have one coordinate node, whenever I try to run parallel, simply run "pipeline benchmark only" two times with different target hosts, I got "race run error", I'm wondering is there anyway can solve this problem?

Thanks

Eric

[dliappis (Dimitrios Liappis)]

Hi Eric,

@jlwysf

Regarding my second question, if "client" is the concurrent session you referred to, how many thread are related in one client, how can I know? Does one client is one process? If we only have one client , is that mean we only execute one concurrent session?

Rally uses the Thespian Actor System which, in turn, is multi-process based. My understanding is that each client corresponds to one Python3 process. Actors themselves are single threaded. See also this explanation. For the record, the instrumentation of all of this is done inside driver.py.

@jlwysf

According to you answer, I haven't seen any index called "rally-*/metrics/" is that because my existing cluster is not provisioned by esrally or they are reside in somewhere else?

Rally stores its metrics in a dedicated Elasticsearch cluster, which is highly recommended anyway so that you can execute queries, use Kibana etc. This is described in the docs here and there's information how to set it up in the advanced configuration section.

@jlwysf

According to this answer I have another question regarding how can I know how many indexing thread I have, does indexing thread is python es client you are talking about earlier?

Indexing threads refer to Elasticsearch's thread pool for indexing (in Java). As per the Elasticsearch docs:

Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.

@jlwysf

Basically I try to run esrally against two existing cluster separately, but I only have one coordinate node, whenever I try to run parallel, simply run "pipeline benchmark only" two times with different target hosts, I got "race run error", I'm wondering is there anyway can solve this problem?

AFAIK you can't use the same Rally coordinator against two different clusters in parallel. It's better that you benchmark your clusters separately and in an isolated manner, also so that the testing results are credible and results from one cluster are not affected by testing the other cluster and vice versa.

Regards,
Dimitris

[danielmitterdorfer]

@jlwysf

Basically I try to run esrally against two existing cluster separately, but I only have one coordinate node, whenever I try to run parallel, simply run "pipeline benchmark only" two times with different target hosts, I got "race run error", I'm wondering is there anyway can solve this problem?

As @dliappis has already answered this is not possible and in fact actively prohibited by Rally (that's why you get that error). The reason is simply to avoid resource contention. Generating load can consume a lot of system resources and we want to avoid that results are skewed by running two benchmarks in parallel.