Bulk index operation with iterations failed

Elastic Stack Elasticsearch
1

I want to use rally to insert many docs to an index for benchmark, e.g. I have a documents.json with 1405 records, I want to insert 1405*1000 to index for benchmark.

{
"operation": "bulk-index",
"iterations": 1000,
"clients": 8,
"detailed-results":true
},

but when I execute the track I got below error

2018-09-28 11:26:15,378 ActorAddr-(T|:65048)/PID:26475 esrally.driver.driver ERROR Could not execute schedule
Traceback (most recent call last):

  File "/usr/local/lib/python3.7/site-packages/esrally/driver/driver.py", line 1327, in iteration_count_based
    yield (next_scheduled, sample_type, percent_completed, runner, params.params())

  File "/usr/local/lib/python3.7/site-packages/esrally/track/params.py", line 527, in params
    return next(self.internal_params)

StopIteration


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/local/lib/python3.7/site-packages/esrally/driver/driver.py", line 985, in __call__
    for expected_scheduled_time, sample_type, percent_completed, runner, params in self.schedule:

RuntimeError: generator raised StopIteration

and actually it only inserted 1405 docs that is iterations:1.

So does iterations not support bulk operation? Or it's caused by some other reason and how to solve it?

P.S.
➜ ~ esrally --pipeline=benchmark-only --target-hosts=127.0.0.1:9200 --track-repository=private --track=companies --challenge=repeat-bulk-index
[INFO] Racing on track [companies], challenge [repeat-bulk-index] and car ['external'] with version [6.2.2].
Running delete [100% done]
Running create [100% done]
Running bulk-index [ 0% done]

[ERROR] Cannot race. Error in load generator [0]
	generator raised StopIteration
2

rally will insert all the documents in the corpora file, and will split it up into bulk requests of the specified size. It will however not send the same data multiple times, which seems to be what you want to do.

1 Like
3

Thanks for reply! But maybe it could have more clear error hint or in Track Reference of iterations could specify it explicitly. And in this scenario, e.g. just want to use rally to import many docs concurrently to es for some test , does exist some other manner could implement it?

4

Hi,

I see that you use Python 3.7 but no released version of Rally supports it yet (we will support it in the upcoming release 1.0.2). The StopIteration error is quite indicative of that fact. I suggest you downgrade to Python 3.6 and retry.

Daniel

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.