Rally fails when running with preserve_benchmark & metricstore

Elastic Stack Elasticsearch
1

Update: Installed 0.4.4. Default run works, preserve_benchmark & metricstore = fatal crash, just like 0.4.3
@danielmitterdorfer - what say you?

rally.ini and rally_out.log posted at gist:
https://gist.github.com/PT-73/41151fb0d75bfce18c614340757f30d9

Original post below for 0.4.3, including log excerpts:
I ran rally 0.4.3 successfully with v5.0, so my installation is working.
Next I wanted to run it and send all benchmark metrics to a separate elasticsearch v5.0 instance (with the metricstore option) and keep the elasticsearch deployment and all benchmark & data directories (with the preserve_benchmark_candidate option). It failed at runtime with errors.
Trying to debug, I ran with persistent setting and without metricstore. Started up fine and commenced with benchmark run. I killed it before completion.

Run command:

esrally --pipeline=from-distribution --distribution-version=5.0.0

Here is my rally.ini, and below that the error lines from rally_out.log

[meta]
config.version = 6

[system]
root.dir = /Users/woodford/.rally/benchmarks
log.root.dir = logs
env.name = Nov-Persist

[provisioning]
local.install.dir = install

[runtime]
java8.home = /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home

[benchmarks]
local.dataset.cache = ${system:root.dir}/data

[reporting]
datastore.type = elasticsearch
datastore.host = localhost
datastore.port = 9200
datastore.secure = False
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

[defaults]
preserve_benchmark_candidate = True

Rally_out.log (excerpt)
Note the 4 "WARNING Unable to get address info" lines appear with successful runs as well as fails, so I'm thinking it's not relevant to this issue.

egrep 'warning|error' /Users/woodford/.rally/benchmarks/races/2016-11-05-17-41-18/Nov-Persist/logs/rally_out.log

2016-11-05 17:41:18,759 root WARNING Unable to get address info for address woodford-mbp13 (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2016-11-05 17:41:18,760 root WARNING Unable to get address info for address woodford-mbp13 (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2016-11-05 17:41:18,761 root WARNING Unable to get address info for address woodford-mbp13 (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 1): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2016-11-05 17:41:18,762 root WARNING Unable to get address info for address woodford-mbp13 (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 1): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
    self._raise_error(response.status, raw_data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)   
    self._raise_error(response.status, raw_data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
2

Hi @Woodford,

This is the issue. I found out yesterday and related to Elasticsearch 5.0. You can either:

  • run from master (see docs on how to setup Rally for development). I'll commit the fix later today.
  • Wait for Rally 0.4.5 (I think I'll reduce the scope of 0.4.5 and release a bugfix release).
  • Use an older version (2.x) of Elasticsearch as metric store

Yes, this is related to the internal actor system we use for Rally (see original issue) and you can ignore it.

Daniel

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.