Parallel Iterations of Bulk

[dliappis (Dimitrios Liappis)]

@David_Samokovlisky

A question for the jinja solution, when I duplicate the same file, Is there any lag\delay between each time it reads the same file again? because continuity is very important for my testings, This way I can save a lot of disc space

When you use a Jinja2 for loop there will be the same (small) delay between tasks to the delay you see between executing explicitly defined operations in a regular track.

@David_Samokovlisky

if I set for each operation a target-throughput , will it slow down the whole thing? because it's a log of timers\sleeps concurrently across 10+ operations and each operation with multiple clients

Tasks inside the parallel element can have their own independent clients and target-throughput and are independent. There is a large number of examples about the parallel element in this part of the documentation that I suggest you take a look at.

@David_Samokovlisky

I have tried playing with the target-throughput , and it seem to be connected\related to bulk-size , I've tried setting the target-throughput to either 1 1000 and not use it at all, and the Median Throughput stays the same, for a parallel 1 bulk operation, with just 1 client.

This sounds normal. target-throughput is not a property that will "accelerate" the execution of bulk by automatically increasing clients; if you've specified 1 client, it will stick to that to achieve the specified target-throughput with 1 client. If target-throughput is smaller that what 1 client can achieve, Rally will pause the schedule as required to honor this, but it won't automatically increase clients to achieve larger throughputs. You need to scale the number of clients/bulk size yourself (and read up on sizing Elasticsearch) if the target-throughput can't be achieved.