Rally 2.3.0 now throwing "should include an inner object describing the mapping" but 2.2.1 was fine?

Elastic Stack Elasticsearch
1

Hey, when I was running rally on 2.2.1 it would run fine without issue but as I'm not specifying a version to use it picked up the latest 2.3.0 and now on run is throwing the following, but I don't get how the templates used are affected? were these changed in the latest version? thanks in advance.

And I'm using Elastic 6.8.15

The race command used for reference.

esrally race --track=geonames --target-hosts=$ips --pipeline=benchmark-only --client-options="basic_auth_user:'$username',basic_auth_password:'$password'"

[INFO] Race id is [6aa14d79-00bd-4438-91ae-3c4fb1cae4fd]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/esrally/metrics.py", line 108, in guarded
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/indices.py", line 642, in put_template
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/transport.py", line 458, in perform_request
    raise e
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/transport.py", line 419, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/base.py", line 330, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.8/dist-packages/esrally/racecontrol.py", line 110, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.8/dist-packages/esrally/racecontrol.py", line 208, in setup
    self.metrics_store = metrics.metrics_store(
  File "/usr/local/lib/python3.8/dist-packages/esrally/metrics.py", line 295, in metrics_store
    store.open(race_id, race_timestamp, track, challenge, selected_car, create=not read_only)
  File "/usr/local/lib/python3.8/dist-packages/esrally/metrics.py", line 867, in open
    self._client.put_template("rally-metrics", self._get_template())
  File "/usr/local/lib/python3.8/dist-packages/esrally/metrics.py", line 60, in put_template
    return self.guarded(self._client.indices.put_template, name=name, body=template)
  File "/usr/local/lib/python3.8/dist-packages/esrally/metrics.py", line 167, in guarded
    raise exceptions.RallyError(msg)
esrally.exceptions.RallyError: A transport error occurred while running the operation [put_template] against your Elasticsearch metrics store on host [10.0.4.4] at port [9200].
2

Hey @ddtombb - the race fails because it cannot setup the correct template in your Metrics store cluster, i.e. where all the benchmarking metrics get sent during a run.

We still support benchmarking 6.x clusters but removed support for 6.x metrics stores in Rally 2.3.0 [1] [2], which is why you're seeing this fail now.

Unfortunately you'll need to either upgrade your existing metrics store, or use an older version of Rally.

Sorry! :frowning:

[1] - Migration Guide - Rally 2.10.0 documentation
[2] - Drop support for Elasticsearch 6.x metrics store by pquentin · Pull Request #1342 · elastic/rally · GitHub

3

Hey Bradley.

thanks for the response that makes sense, I've tested using 2.21 but now run into a random error without much context

wget https://github.com/elastic/rally/releases/download/2.2.0/esrally-dist-linux-2.2.0.tar.gz
tar -zxvf esrally-dist-linux-2.2.0.tar.gz
cd esrally-dist-2.2.0/
sudo -H ./install.sh
esrally --version

----------------------------------
[INFO] SUCCESS (took 2608 seconds)
----------------------------------
##[debug]$LASTEXITCODE: 0

##[debug]Exit code 0 received from tool '/usr/bin/pwsh'
##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
##[error]--2021-12-01 17:39:24--  https://github.com/elastic/rally/releases/download/2.2.0/esrally-dist-linux-2.2.0.tar.gz
140.82.121.4
connected.
302 Found
Location: https://github-releases.githubusercontent.com/47140131/36003600-12eb-15eb-0tf6-36003600?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVAAIWNJYAX%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211201T173924Z&X-Amz-Expires=300&X-Amz-Signature=f99aaa66b83d9bf254959adaa66b83d9bf254959ad87d6bba21e81ee7&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=47140131&response-content-disposition=attachment%3B%20filename%3Desrally-dist-linux-2.2.0.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-12-01 17:39:24--  https://github-releases.githubusercontent.com/47140131/fe2480600-480b62-480b62-9ef6-30dfe2480b62?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211201T173924Z&X-Amz-Expires=300&X-Amz-Signature=f99aaa66b83d9bf254959adaa66b83d9bf254959ad87d6bba21e81ee7&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=47140131&response-content-disposition=attachment%3B%20filename%3Desrally-dist-linux-2.2.0.tar.gz&response-content-type=application%2Foctet-stream
185.199.111.154, 185.199.110.154, 185.199.108.154, ...
connected.
200 OK
 (12M) [application/octet-stream]
##[debug]Processed: ##vso[task.issue type=error;]--2021-12-01 17:39:24--  https://github.com/elastic/rally/releases/download/2.2.0/esrally-dist-linux-2.2.0.tar.gz%0A140.82.121.4%0Aconnected.%0A302 Found%0ALocation: https://github-releases.githubusercontent.com/47140131/7ac03600-a38f-11eb-9ef6-30dfe2480b62?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWVEH53AVEH53A%AZPVEH53A11201%AZP252Fus-east-1%AZPVEH53A3%AZVEH53AFaws4_request&X-Amz-Date=20211VEH53A924Z&X-Amz-Expires=300&X-Amz-Signature=f99aaa66b83d9bf254959adaa66b83d9bf254959ad87d6bba21e81ee7&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=47140131&response-content-disposition=attachment%AZP253B%AZP2520filename%AZP253Desrally-dist-linux-2.2.0.tar.gz&response-content-type=application%AZP252Foctet-stream [following]%0A--2021-12-01 17:39:24--  
##[debug]task result: Failed
##[error]Script has output to stderr. Failing as failOnStdErr is set to true.
4

I can see what you are running this from within Powershell.

Rally hasn't been tested again Windows installation (native or WSL).
Could you please try the offline installation instructions from within a Linux distribution (native or VM)?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.