Running benchmark-only error

Elastic Stack Elasticsearch
1

I run "esrally --track=geonames --offline --pipeline=benchmark-only --target-hosts=xxx.xxx.xxx.xxx:9200"
have error:

2016-10-18 11:48:21,989 elasticsearch WARNING PUT /geonames/_mapping/type [status:400 request:1.315s]
2016-10-18 11:48:21,995 rally.telemetry INFO Benchmark stop
2016-10-18 11:48:21,995 rally.telemetry INFO Gathering nodes stats
2016-10-18 11:48:26,414 rally.telemetry INFO Gathering indices stats
2016-10-18 11:48:32,531 rally.telemetry WARNING Could not determine metric [segments_points_memory_in_bytes] at path [segments,points_memory_in_bytes].
2016-10-18 11:48:32,558 root ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 364, in run
    pipeline()
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 60, in __call__
    step()
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 29, in __call__
    self.command(self.ctx)
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 188, in benchmark_external
    raise exceptions.RallyError("Driver has returned no metrics but instead [%s]. Terminating race without result." % str(completed))
esrally.exceptions.RallyError: Driver has returned no metrics but instead [Poison<<esrally.driver.StartBenchmark object at 0x7fe3632d5b70>>]. Terminating race without result.

then, I use ”esrally --track=geonames --offline --pipeline=benchmark-only --distribution-version=2.3.2 --target-hosts=xxx.xxx.xxx.xxx:9200”，
but have error：

2016-10-18 12:20:29,450 rally.driver ERROR Cluster did not reach status [green]. Last reached status: [green]
2016-10-18 12:20:29,454 rally.telemetry INFO Benchmark stop
2016-10-18 12:20:29,454 rally.telemetry INFO Gathering nodes stats
2016-10-18 12:20:37,800 rally.telemetry INFO Gathering indices stats
2016-10-18 12:20:43,231 rally.telemetry WARNING Could not determine metric [segments_points_memory_in_bytes] at path [segments,points_memory_in_bytes].
2016-10-18 12:20:43,257 root ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 364, in run
    pipeline()
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 60, in __call__
    step()
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 29, in __call__
    self.command(self.ctx)
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 188, in benchmark_external
    raise exceptions.RallyError("Driver has returned no metrics but instead [%s]. Terminating race without result." % str(completed))
esrally.exceptions.RallyError: Driver has returned no metrics but instead [Poison<<esrally.driver.StartBenchmark object at 0x7f4669f1b240>>]. Terminating race without result.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/esrally/rally.py", line 413, in dispatch_sub_command
    racecontrol.run(cfg)
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 367, in run
    raise exceptions.RallyError("This race ended early with a fatal crash. For details please see the logs.").with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 364, in run
    pipeline()
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 60, in __call__
    step()
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 29, in __call__
    self.command(self.ctx)
  File "/usr/local/lib/python3.5/site-packages/esrally/racecontrol.py", line 188, in benchmark_external
    raise exceptions.RallyError("Driver has returned no metrics but instead [%s]. Terminating race without result." % str(completed))
esrally.exceptions.RallyError: This race ended early with a fatal crash. For details please see the logs.
2

Hi @duramen,

It seems you're running an older version of Rally. Can you please update to the latest version of Rally and retry? I don't expect that error is gone but the error reporting got better in these cases. There is also no need anymore to specify --distribution-version when you use --pipeline=benchmark-only as Rally will autodetect the version.

Also --offline shouldn't be necessary.

Daniel

3

I have updated rally 0.4.3,but have the same error:
2016-10-19 02:14:15,780 rally.driver ERROR Cluster did not reach status [green]. Last reached status: [green]

4

Hi @duramen,

I checked the source code. Rally checks if the cluster health is green and that there are no relocating shards. So I am quite sure that you have relocating shards. You can check that yourself with the cat/_shards API. E.g.:

curl http://YOUR_CLUSTER_IP:9200/_cat/shards | fgrep RELO

This should show your relocating shards.

You might wonder why Rally refuses to run a benchmark against your cluster in this case. The idea behind this check is that you should not run a benchmark if your cluster is busy, otherwise you will not get representative results. And if shards are relocating in this moment, it takes away I/O capacity for the benchmark.

I've created https://github.com/elastic/rally/issues/171 to improve the error message in this case.

Daniel

5

hi @danielmitterdorfer,

I have deleted the relocating shards and running ok .

but I don't see about disk(disk_io_write_bytes...) in the report.

https://esrally.readthedocs.io/en/latest/metrics.html#metric-keys

I need check disk IO。

and another question：how can print report in command-line by esrally command？

6

Hi @duramen,

ok, great to hear that it works now.

In the report this value is shown as "Totally written". The metric key is just used internally and is mainly interesting if you have configured an Elasticsearch metrics store and want to understand how to visualize metrics in Kibana.

There is no separate command to show the report (again) on the command line. However, the report is also written to the log file that is stored in ~/.rally/benchmarks/races/$TIMESTAMP_OF_YOUR_BENCHMARK/logs/ (just unzip the logs and it's right there at the end of the rally_out.log).

If you want to safe the report as a file you can store it (with --report-file) either as Markdown or CSV (with --report-format).

Daniel