Esrally run failed for "Cannot find documents-2.json.bz2"

Jessica254 · June 27, 2018, 5:38am

No, I don't think it's behind any proxy. Or could you provide the file through email or other way please?

danielmitterdorfer · June 28, 2018, 4:52am

One last option is to to use the HTTPS link (either via curl or a browser):

curl -o documents-2.json.bz2 https://s3.amazonaws.com/benchmarks.elasticsearch.org/corpora/geonames/documents-2.json.bz2

Can you please try this and report whether this works for you?

Unfortunately it is not possible to provide these files in an alternative way (e.g. via email). However, these tracks are anyway only meant to get you started. So if the https link does not work either, I think it makes more sense that you create your own track based on your data.

Jessica254 · June 28, 2018, 6:38am

Aha, it seems OK:
[root@hdh154 /home/.rally/benchmarks/data/geonames]# md5sum documents-2.json.bz2
c6fbf5e7b20c3c46f4cd6ab8385a9cb7 documents-2.json.bz2

But, why the test data is 0

_______             __   _____

/ () __ / / / /_ ______
/ /_ / / / `/ / / / / / _
/ / / / / / / // / / / / // // / / / /
// /// //_,// /__/_/_// ___/

Lap	Metric	Task	Value	Unit
All	Indexing time		0	min
All	Indexing throttle time		0	min
All	Merge time		0	min
All	Refresh time		0	min
All	Flush time		0	min
All	Merge throttle time		0	min
All	Total Young Gen GC		3.772	s
All	Total Old Gen GC		0	s
All	error rate	index-append	0	%

[WARNING] No throughput metrics available for [index-append]. Likely cause: The benchmark ended already during warmup.

[INFO] SUCCESS (took 257 seconds)

danielmitterdorfer · June 29, 2018, 5:10am

I really wonder what prevented the http version to work.

The output at the end tells the reason:

This track has a built-in warmup time-period of two minutes (120 seconds) and only after that time period Rally will start to consider measurements for the final report. However, in your benchmark the operation index-append took less than those two minutes and hence it does not show any numbers.

For playing around you can lower those numbers in the track (i.e. in ~/.rally/benchmarks/tracks/default/geonames/challenge/default.json). Find all lines where it says:

"warmup-time-period": 120

and reduce that number (e.g. to 60).

Jessica254 · June 29, 2018, 6:59am

it can work now, thank you so much:kissing_heart:

_______             __   _____

/ () __ / / / /_ ______
/ /_ / / / `/ / / / / / _
/ / / / / / / // / / / / // // / / / /
// /// //_,// /__/_/_// ___/

Lap	Metric	Task	Value	Unit
All	Indexing time		0	min
All	Indexing throttle time		0	min
All	Merge time		0	min
All	Refresh time		0	min
All	Flush time		0	min
All	Merge throttle time		0	min
All	Total Young Gen GC		3.559	s
All	Total Old Gen GC		0	s
All	Min Throughput	index-append	116890	docs/s
All	Median Throughput	index-append	118683	docs/s
All	Max Throughput	index-append	119238	docs/s
All	50th percentile latency	index-append	281.031	ms
All	90th percentile latency	index-append	347.315	ms
All	99th percentile latency	index-append	400.704	ms
All	99.9th percentile latency	index-append	444.735	ms
All	100th percentile latency	index-append	524.444	ms
All	50th percentile service time	index-append	281.031	ms
All	90th percentile service time	index-append	347.315	ms
All	99th percentile service time	index-append	400.704	ms
All	99.9th percentile service time	index-append	444.735	ms
All	100th percentile service time	index-append	524.444	ms
All	error rate	index-append	100	%

[INFO] SUCCESS (took 152 seconds)

danielmitterdorfer · June 29, 2018, 7:22am

Hi @Jessica254,

that looks much better, yes! But there is this:

It indicates that there is some systematic problem writing data to the cluster. By default Rally will continue on errors and only report the error rate. In this case it seems it had trouble indexing data for some reason. Can you please add the parameter --on-error=abort and rerun? With that parameter added, Rally will abort the benchmark on the first error and tell you what's wrong. Maybe you are missing a file or something else is inconsistent.

Daniel

Jessica254 · June 29, 2018, 8:40am

It's the error:
[INFO] Racing on track [geonames], challenge [append-no-conflicts-index-only] and car ['external'] with version [2.3.5].

Running delete-index [100% done]
Running create-index [ 0% done]
[ERROR] Cannot race. Error in load generator [0]
Request returned an error. Error type: transport, Description: mapper_parsing_exception ({'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}], 'type': 'mapper_parsing_exception', 'reason': 'Failed to parse mapping [type]: No handler for type [text] declared on field [admin1_code]', 'caused_by': {'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}}, 'status': 400})

danielmitterdorfer · June 29, 2018, 8:50am

It looks as if Rally did not checkout the correct branch in the rally-tracks repo. Can you please execute the following?

cd ~/.rally/benchmarks/tracks/default
git status

If it does not say On branch 2, can you please do:

rm ~/.rally/logs/rally.log
# then rerun the benchmark

and paste the contents of ~/.rally/logs/rally.log here?

Jessica254 · July 3, 2018, 7:22am

sorry for reply late, env is not available for rally.
Here's the err log:

[root@hdh154 ~/.rally/logs]# cat rally-actor-messages.log
2018-07-03 07:32:29,434 -not-actor-/PID:21112 root INFO ActorSystem Logging Shutdown

[root@hdh154 ~/.rally/logs]# tail -n 10 rally_out_20180703T073216Z.log
raise e
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 374, in run
pipeline(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 63, in call
self.target(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 322, in benchmark_only
return race(cfg, external=True)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 276, in race
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: ('Error in load generator [0]', Request returned an error. Error type: transport, Description: mapper_parsing_exception ({'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}], 'type': 'mapper_parsing_exception', 'reason': 'Failed to parse mapping [type]: No handler for type [text] declared on field [admin1_code]', 'caused_by': {'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}}, 'status': 400}))

danielmitterdorfer · July 3, 2018, 7:46am

Hi,

no worries. However, I have few questions:

What was the output of this command?

cd ~/.rally/benchmarks/tracks/default
git status

I'd also need the full file contents of the log file because I suspect that for some reason Rally is not checking out the correct branch in the track repository (i.e. in ~/.rally/benchmarks/tracks/default) because apparently it should checkout branch 2 in order to have the configuration for Elasticsearch 2.x available. If I only see the last few lines I cannot understand what Rally has done before.

Daniel

Jessica254 · July 3, 2018, 8:10am

git result:
[root@hdh154 ~/.rally/benchmarks/tracks/default]# git status
On branch master
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

modified:   geonames/challenges/default.json

Untracked files:
(use "git add ..." to include in what will be committed)

geonames/__pycache__/

and there's 2 log files under log:
[root@hdh154 ~/.rally/logs]# ll
total 64
-rw-r--r-- 1 root root 85 Jul 3 15:32 rally-actor-messages.log
-rw-r--r-- 1 root root 57743 Jul 3 15:32 rally_out_20180703T073216Z.log

I have put full contents for the first file, and for the second, it's too long to paste on the reply.
maybe I can separate it, for 1st part:
2018-07-03 07:32:16,827 PID:21105 rally.main INFO OS [posix.uname_result(sysname='Linux', nodename='hdh154', release='2.6.32-573.el6.x86_64', version='#1 SMP Thu Jul 23 15:44:03 UTC 2015', machine='x86_64')]
2018-07-03 07:32:16,827 PID:21105 rally.main INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50725616, name='cpython', version=sys.version_info(major=3, minor=6, micro=2, releaselevel='final', serial=0))]
2018-07-03 07:32:16,827 PID:21105 rally.main INFO Rally version [0.11.0]
2018-07-03 07:32:16,827 PID:21105 rally.main INFO Command line arguments: Namespace(advanced_config=False, assume_defaults=False, car='defaults', car_params='', challenge='append-no-conflicts-index-only', client_options='timeout:60', configuration_name=None, distribution_repository='release', distribution_version='', effective_start_date=None, elasticsearch_plugins='', enable_driver_profiling=False, include_tasks=None, java_home=None, keep_cluster_running=False, laps=1, load_driver_hosts='localhost', logging='file', offline=True, on_error='abort', pipeline='benchmark-only', plugin_params='', preserve_install='False', quiet=False, report_file='', report_format='markdown', revision='current', runtime_java_home=None, show_in_report='available', subcommand=None, target_hosts='10.3.68.178:9200', team_path=None, team_repository='default', telemetry='', telemetry_params='', test_mode=False, track='geonames', track_params='', track_path=None, track_repository='default', user_tag='')
2018-07-03 07:32:16,828 PID:21105 rally.net INFO Rally connects directly to the Internet (no proxy support).
2018-07-03 07:32:16,971 PID:21105 rally.process INFO Skipping myself (PID [21105]).
2018-07-03 07:32:16,974 PID:21105 rally.main INFO Actor system already running locally? [False]
2018-07-03 07:32:16,974 PID:21105 rally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2018-07-03 07:32:17,14 PID:21111 root INFO ++++ Actor System gen (3, 9) started, admin @ ActorAddr-(T|:1900)
2018-07-03 07:32:17,37 PID:21105 rally.racecontrol INFO User specified pipeline [benchmark-only].
2018-07-03 07:32:17,38 PID:21105 rally.racecontrol INFO Using configured hosts [{'host': '10.3.68.178', 'port': 9200}]
2018-07-03 07:32:17,39 PID:21105 rally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2018-07-03 07:32:17,40 PID:21111 rally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 2, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1530603137001'}] match requirements [{'coordinator': True}].
2018-07-03 07:32:17,41 PID:21111 rally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 2, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1530603137001'}] match requirements [{'coordinator': True}].
2018-07-03 07:32:17,57 PID:21113 rally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 2, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1530603137001'}] match requirements [{'coordinator': True}].
2018-07-03 07:32:17,59 PID:21113 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,60 PID:21113 rally.client INFO SSL support: off
2018-07-03 07:32:17,60 PID:21113 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,61 PID:21113 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,68 PID:21113 rally.racecontrol INFO Automatically derived distribution version [2.3.5]
2018-07-03 07:32:17,81 PID:21113 rally.repo INFO Checking out [master] in [/root/.rally/benchmarks/tracks/default] for distribution version [2.3.5].
2018-07-03 07:32:17,99 PID:21113 rally.track INFO Reading track specification file [/root/.rally/benchmarks/tracks/default/geonames/track.json].
2018-07-03 07:32:17,129 PID:21113 rally.track INFO Final rendered track for '/root/.rally/benchmarks/tracks/default/geonames/track.json':

Jessica254 · July 3, 2018, 8:13am

2nd part is the configuaration file content, do you need to read it?
the 3rd part:
2018-07-03 07:32:17,139 PID:21113 rally.track INFO Loading template [definition for index geonames in index.json].
2018-07-03 07:32:17,144 PID:21113 rally.metrics INFO Creating in-memory metrics store
2018-07-03 07:32:17,144 PID:21113 rally.metrics INFO Opening metrics store for trial timestamp=[20180703T073216Z], track=[geonames], challenge=[append-no-conflicts-index-only], car=[['external']]
2018-07-03 07:32:17,144 PID:21113 rally.metrics INFO Creating file race store
2018-07-03 07:32:17,145 PID:21113 rally.racecontrol INFO Asking mechanic to start the engine.
2018-07-03 07:32:17,148 PID:21114 rally.mechanic INFO Received signal from race control to start engine.
2018-07-03 07:32:17,148 PID:21114 rally.metrics INFO Opening metrics store for trial timestamp=[20180703T073216Z], track=[geonames], challenge=[append-no-conflicts-index-only], car=[['external']]
2018-07-03 07:32:17,149 PID:21114 rally.mechanic INFO Cluster will not be provisioned by Rally.
2018-07-03 07:32:17,149 PID:21114 rally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 2, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1530603137001'}] match requirements [{'coordinator': True}].
2018-07-03 07:32:17,165 PID:21119 rally.mechanic INFO Connecting to externally provisioned nodes on [[{'host': '10.3.68.178', 'port': 9200}]].
2018-07-03 07:32:17,171 PID:21119 rally.metrics INFO Opening metrics store for trial timestamp=[20180703T073216Z], track=[geonames], challenge=[append-no-conflicts-index-only], car=[['external']]
2018-07-03 07:32:17,171 PID:21119 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,171 PID:21119 rally.client INFO SSL support: off
2018-07-03 07:32:17,171 PID:21119 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,172 PID:21119 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,209 PID:21114 rally.actor INFO [1] of [1] child actors have responded for transition from [starting] to [nodes_started].
2018-07-03 07:32:17,209 PID:21114 rally.actor INFO All [1] child actors have responded. Transitioning now from [starting] to [nodes_started].
2018-07-03 07:32:17,210 PID:21114 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,210 PID:21114 rally.client INFO SSL support: off
2018-07-03 07:32:17,210 PID:21114 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,211 PID:21114 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,211 PID:21114 rally.launcher INFO All cluster nodes have successfully started. Checking if REST API is available.
2018-07-03 07:32:17,217 PID:21114 rally.launcher INFO REST API is available. Attaching telemetry devices to cluster.
2018-07-03 07:32:17,240 PID:21114 rally.launcher INFO Telemetry devices are now attached to the cluster.
2018-07-03 07:32:17,240 PID:21114 rally.actor INFO Transitioning from [nodes_started] to [apply_meta_info].
2018-07-03 07:32:17,244 PID:21114 rally.actor INFO [1] of [1] child actors have responded for transition from [apply_meta_info] to [cluster_started].
2018-07-03 07:32:17,244 PID:21114 rally.actor INFO All [1] child actors have responded. Transitioning now from [apply_meta_info] to [cluster_started].
2018-07-03 07:32:17,247 PID:21113 rally.racecontrol INFO Mechanic has started engine successfully.
2018-07-03 07:32:17,247 PID:21113 rally.racecontrol INFO Starting lap [1/1]
2018-07-03 07:32:17,248 PID:21113 rally.racecontrol INFO Telling mechanic of benchmark start.
2018-07-03 07:32:17,249 PID:21113 rally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 2, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1530603137001'}] match requirements [{'coordinator': True}].

Jessica254 · July 3, 2018, 8:14am

the 4th part:
2018-07-03 07:32:17,250 PID:21114 rally.actor INFO Transitioning from [cluster_started] to [benchmark_starting].
2018-07-03 07:32:17,254 PID:21114 rally.actor INFO [1] of [1] child actors have responded for transition from [benchmark_starting] to [benchmark_started].
2018-07-03 07:32:17,254 PID:21114 rally.actor INFO All [1] child actors have responded. Transitioning now from [benchmark_starting] to [benchmark_started].
2018-07-03 07:32:17,253 PID:21113 rally.racecontrol INFO Telling driver to start benchmark.
2018-07-03 07:32:17,260 PID:21114 rally.telemetry WARNING Could not determine value at path [merges,total_time_in_millis]. Returning default value [0]
2018-07-03 07:32:17,261 PID:21114 rally.telemetry WARNING Could not determine value at path [merges,total_throttled_time_in_millis]. Returning default value [0]
2018-07-03 07:32:17,261 PID:21114 rally.telemetry WARNING Could not determine value at path [indexing,index_time_in_millis]. Returning default value [0]
2018-07-03 07:32:17,261 PID:21114 rally.telemetry WARNING Could not determine value at path [indexing,throttle_time_in_millis]. Returning default value [0]
2018-07-03 07:32:17,261 PID:21114 rally.telemetry WARNING Could not determine value at path [refresh,total_time_in_millis]. Returning default value [0]
2018-07-03 07:32:17,261 PID:21114 rally.telemetry WARNING Could not determine value at path [flush,total_time_in_millis]. Returning default value [0]
2018-07-03 07:32:17,266 PID:21113 rally.racecontrol INFO BenchmarkActor received unknown message [<esrally.mechanic.mechanic.BenchmarkStarted object at 0x7f8302084390>] (ignoring).
2018-07-03 07:32:17,268 PID:21120 rally.metrics INFO Creating in-memory metrics store
2018-07-03 07:32:17,268 PID:21120 rally.metrics INFO Opening metrics store for trial timestamp=[20180703T073216Z], track=[geonames], challenge=[append-no-conflicts-index-only], car=[['external']]
2018-07-03 07:32:17,269 PID:21120 rally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 2, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1530603137001'}] match requirements [{'coordinator': True}].
2018-07-03 07:32:17,290 PID:21121 rally.driver INFO Preparing track [geonames]
2018-07-03 07:32:17,303 PID:21121 rally.repo INFO Checking out [master] in [/root/.rally/benchmarks/tracks/default] for distribution version [2.3.5].
2018-07-03 07:32:17,318 PID:21121 rally.track INFO Resolved data root directory for document corpus [geonames] in track [geonames] to ['/home/.rally/benchmarks/data/geonames'].
2018-07-03 07:32:17,319 PID:21121 rally.utils.io INFO Skipping creation of file offset table at [/home/.rally/benchmarks/data/geonames/documents-2.json.offset] as it is still valid.
2018-07-03 07:32:17,321 PID:21120 rally.actor INFO [1] of [1] child actors have responded for transition from [init] to [None].
2018-07-03 07:32:17,321 PID:21120 rally.actor INFO All [1] child actors have responded. Transitioning now from [init] to [None].
2018-07-03 07:32:17,322 PID:21120 rally.driver INFO Benchmark is about to start.
2018-07-03 07:32:17,322 PID:21120 rally.driver INFO Resetting relative time of request metrics store.
2018-07-03 07:32:17,323 PID:21120 rally.driver INFO Benchmark consists of [5] steps executed by (at most) [8] clients as specified by the allocation matrix:
[[JoinPoint(0), TaskAllocation [0/1] for delete-index, JoinPoint(1), TaskAllocation [0/1] for create-index, JoinPoint(2), TaskAllocation [0/1] for check-cluster-health, JoinPoint(3), TaskAllocation [0/8] for index-append, JoinPoint(4), TaskAllocation [0/1] for force-merge, JoinPoint(5)], [JoinPoint(0), None, JoinPoint(1), None, JoinPoint(2), None, JoinPoint(3), TaskAllocation [1/8] for index-append, JoinPoint(4), None, JoinPoint(5)], [JoinPoint(0), None, JoinPoint(1), None, JoinPoint(2), None, JoinPoint(3), TaskAllocation [2/8] for index-append, JoinPoint(4), None, JoinPoint(5)], [JoinPoint(0), None, JoinPoint(1), None, JoinPoint(2), None, JoinPoint(3), TaskAllocation [3/8] for index-append, JoinPoint(4), None, JoinPoint(5)], [JoinPoint(0), None, JoinPoint(1), None, JoinPoint(2), None, JoinPoint(3), TaskAllocation [4/8] for index-append, JoinPoint(4), None, JoinPoint(5)], [JoinPoint(0), None, JoinPoint(1), None, JoinPoint(2), None, JoinPoint(3), TaskAllocation [5/8] for index-append, JoinPoint(4), None, JoinPoint(5)], [JoinPoint(0), None, JoinPoint(1), None, JoinPoint(2), None, JoinPoint(3), TaskAllocation [6/8] for index-append, JoinPoint(4), None, JoinPoint(5)], [JoinPoint(0), None, JoinPoint(1), None, JoinPoint(2), None, JoinPoint(3), TaskAllocation [7/8] for index-append, JoinPoint(4), None, JoinPoint(5)]]
2018-07-03 07:32:17,323 PID:21120 rally.driver INFO Allocating load generator [0] on [localhost]
2018-07-03 07:32:17,323 PID:21120 rally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 2, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1530603137001'}] match requirements [{'coordinator': True}].

Jessica254 · July 3, 2018, 8:15am

the 5th part:
2018-07-03 07:32:17,348 PID:21120 rally.driver INFO Starting load generator [7].
2018-07-03 07:32:17,351 PID:21120 rally.driver INFO A track preparator has exited.
2018-07-03 07:32:17,355 PID:21126 rally.driver INFO LoadGenerator[0] is about to start.
2018-07-03 07:32:17,357 PID:21120 rally.driver INFO A track preparator has exited.
2018-07-03 07:32:17,358 PID:21127 rally.driver INFO LoadGenerator[1] is about to start.
2018-07-03 07:32:17,360 PID:21126 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,360 PID:21126 rally.client INFO SSL support: off
2018-07-03 07:32:17,360 PID:21126 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,360 PID:21129 rally.driver INFO LoadGenerator[3] is about to start.
2018-07-03 07:32:17,360 PID:21126 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,361 PID:21127 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,361 PID:21127 rally.client INFO SSL support: off
2018-07-03 07:32:17,361 PID:21127 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,361 PID:21126 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,361 PID:21127 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,362 PID:21127 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,363 PID:21128 rally.driver INFO LoadGenerator[2] is about to start.
2018-07-03 07:32:17,363 PID:21129 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,363 PID:21129 rally.client INFO SSL support: off
2018-07-03 07:32:17,363 PID:21126 rally.driver INFO LoadGenerator[0] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,363 PID:21129 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,364 PID:21129 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,364 PID:21127 rally.driver INFO LoadGenerator[1] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,364 PID:21130 rally.driver INFO LoadGenerator[4] is about to start.
2018-07-03 07:32:17,365 PID:21128 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,365 PID:21129 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,365 PID:21128 rally.client INFO SSL support: off
2018-07-03 07:32:17,365 PID:21128 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,366 PID:21128 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,367 PID:21128 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,367 PID:21130 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,367 PID:21130 rally.client INFO SSL support: off

Jessica254 · July 3, 2018, 8:16am

the 6th part:
2018-07-03 07:32:17,367 PID:21120 rally.driver INFO [1/8] drivers reached join point [0/5].
2018-07-03 07:32:17,367 PID:21130 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,368 PID:21130 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,368 PID:21120 rally.driver INFO [2/8] drivers reached join point [0/5].
2018-07-03 07:32:17,368 PID:21129 rally.driver INFO LoadGenerator[3] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,368 PID:21130 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,369 PID:21128 rally.driver INFO LoadGenerator[2] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,369 PID:21131 rally.driver INFO LoadGenerator[5] is about to start.
2018-07-03 07:32:17,370 PID:21120 rally.driver INFO [3/8] drivers reached join point [0/5].
2018-07-03 07:32:17,370 PID:21132 rally.driver INFO LoadGenerator[6] is about to start.
2018-07-03 07:32:17,370 PID:21130 rally.driver INFO LoadGenerator[4] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,370 PID:21120 rally.driver INFO [4/8] drivers reached join point [0/5].
2018-07-03 07:32:17,371 PID:21120 rally.driver INFO [5/8] drivers reached join point [0/5].
2018-07-03 07:32:17,371 PID:21131 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,372 PID:21131 rally.client INFO SSL support: off
2018-07-03 07:32:17,372 PID:21131 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,372 PID:21133 rally.driver INFO LoadGenerator[7] is about to start.
2018-07-03 07:32:17,372 PID:21131 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,373 PID:21131 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,374 PID:21132 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,374 PID:21132 rally.client INFO SSL support: off
2018-07-03 07:32:17,374 PID:21132 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,375 PID:21132 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,375 PID:21131 rally.driver INFO LoadGenerator[5] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,375 PID:21133 rally.client INFO Creating ES client connected to [{'host': '10.3.68.178', 'port': 9200}] with options [{'timeout': 60}]
2018-07-03 07:32:17,375 PID:21132 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,375 PID:21133 rally.client INFO SSL support: off
2018-07-03 07:32:17,375 PID:21133 rally.client INFO HTTP basic authentication: off
2018-07-03 07:32:17,376 PID:21133 rally.client INFO HTTP compression: off
2018-07-03 07:32:17,376 PID:21120 rally.driver INFO [6/8] drivers reached join point [0/5].
2018-07-03 07:32:17,377 PID:21133 rally.modules INFO Loading component [geonames] from [/root/.rally/benchmarks/tracks/default/geonames]
2018-07-03 07:32:17,377 PID:21132 rally.driver INFO LoadGenerator[6] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,378 PID:21133 rally.driver INFO LoadGenerator[7] reached join point [JoinPoint(0)].
2018-07-03 07:32:17,379 PID:21120 rally.driver INFO [7/8] drivers reached join point [0/5].
2018-07-03 07:32:17,380 PID:21120 rally.driver INFO [8/8] drivers reached join point [0/5].
2018-07-03 07:32:17,380 PID:21120 rally.driver INFO All drivers completed their operations until join point [0/5].
2018-07-03 07:32:17,380 PID:21120 rally.driver INFO Postprocessing samples...
2018-07-03 07:32:17,380 PID:21120 rally.metrics INFO Compression changed size of metric store from [64] bytes to [47] bytes
2018-07-03 07:32:17,381 PID:21120 rally.driver INFO Scheduling next task for client id [0] at their timestamp [8897249.731161] (master timestamp [8897249.734700])
2018-07-03 07:32:17,382 PID:21120 rally.driver INFO Scheduling next task for client id [1] at their timestamp [8897249.731402] (master timestamp [8897249.734700])
2018-07-03 07:32:17,382 PID:21113 rally.racecontrol INFO Task has finished.
2018-07-03 07:32:17,382 PID:21113 rally.racecontrol INFO Bulk adding request metrics to metrics store.
2018-07-03 07:32:17,382 PID:21113 rally.metrics INFO Restoring in-memory representation of metrics store.
2018-07-03 07:32:17,382 PID:21126 rally.driver INFO LoadGenerator[0] is continuing its work at task index [1] on [8897249.731161], that is in [0:00:00.994983].
2018-07-03 07:32:17,383 PID:21120 rally.driver INFO Scheduling next task for client id [2] at their timestamp [8897249.733004] (master timestamp [8897249.734700])

Jessica254 · July 3, 2018, 8:16am

the 7th part:
2018-07-03 07:32:18,379 PID:21126 rally.driver INFO LoadGenerator[0] starts driving now.
2018-07-03 07:32:18,379 PID:21126 rally.driver INFO LoadGenerator[0] is executing [delete-index].
2018-07-03 07:32:18,379 PID:21127 rally.driver INFO LoadGenerator[1] starts driving now.
2018-07-03 07:32:18,380 PID:21127 rally.driver INFO LoadGenerator[1] reached join point [JoinPoint(1)].
2018-07-03 07:32:18,380 PID:21126 rally.driver INFO Choosing [deterministic scheduler] for [delete-index].
2018-07-03 07:32:18,380 PID:21126 rally.driver INFO Creating iteration-count based schedule with [deterministic] distribution for [delete-index] with [0] warmup iterations and [1] iterations.
2018-07-03 07:32:18,381 PID:21120 rally.driver INFO Resetting relative time of request metrics store.
2018-07-03 07:32:18,381 PID:21128 rally.driver INFO LoadGenerator[2] starts driving now.
2018-07-03 07:32:18,381 PID:21132 rally.driver INFO LoadGenerator[6] starts driving now.
2018-07-03 07:32:18,381 PID:21131 rally.driver INFO LoadGenerator[5] starts driving now.
2018-07-03 07:32:18,381 PID:21128 rally.driver INFO LoadGenerator[2] reached join point [JoinPoint(1)].
2018-07-03 07:32:18,381 PID:21130 rally.driver INFO LoadGenerator[4] starts driving now.
2018-07-03 07:32:18,381 PID:21133 rally.driver INFO LoadGenerator[7] starts driving now.
2018-07-03 07:32:18,382 PID:21131 rally.driver INFO LoadGenerator[5] reached join point [JoinPoint(1)].
2018-07-03 07:32:18,382 PID:21132 rally.driver INFO LoadGenerator[6] reached join point [JoinPoint(1)].
2018-07-03 07:32:18,382 PID:21133 rally.driver INFO LoadGenerator[7] reached join point [JoinPoint(1)].
2018-07-03 07:32:18,382 PID:21130 rally.driver INFO LoadGenerator[4] reached join point [JoinPoint(1)].
2018-07-03 07:32:18,382 PID:21129 rally.driver INFO LoadGenerator[3] starts driving now.
2018-07-03 07:32:18,383 PID:21129 rally.driver INFO LoadGenerator[3] reached join point [JoinPoint(1)].
2018-07-03 07:32:18,384 PID:21120 rally.driver INFO [1/8] drivers reached join point [1/5].
2018-07-03 07:32:18,385 PID:21120 rally.driver INFO [2/8] drivers reached join point [1/5].
2018-07-03 07:32:18,386 PID:21114 rally.mechanic INFO Resetting relative time of cluster system metrics store.
2018-07-03 07:32:18,387 PID:21120 rally.driver INFO [3/8] drivers reached join point [1/5].
2018-07-03 07:32:18,387 PID:21120 rally.driver INFO [4/8] drivers reached join point [1/5].
2018-07-03 07:32:18,388 PID:21120 rally.driver INFO [5/8] drivers reached join point [1/5].
2018-07-03 07:32:18,388 PID:21120 rally.driver INFO [6/8] drivers reached join point [1/5].
2018-07-03 07:32:18,388 PID:21119 rally.mechanic INFO Resetting relative time of system metrics store on host [[{'host': '10.3.68.178', 'port': 9200}]].
2018-07-03 07:32:18,389 PID:21120 rally.driver INFO [7/8] drivers reached join point [1/5].
2018-07-03 07:32:23,386 PID:21126 rally.driver INFO LoadGenerator[0] is ready for the next task.
2018-07-03 07:32:23,386 PID:21126 rally.driver INFO LoadGenerator[0] reached join point [JoinPoint(1)].
2018-07-03 07:32:23,389 PID:21120 rally.driver INFO [8/8] drivers reached join point [1/5].
2018-07-03 07:32:23,390 PID:21120 rally.driver INFO All drivers completed their operations until join point [1/5].
2018-07-03 07:32:23,390 PID:21120 rally.driver INFO Postprocessing samples...
2018-07-03 07:32:23,390 PID:21120 rally.driver INFO Storing latency and service time took [0.000241] seconds.
2018-07-03 07:32:23,391 PID:21120 rally.driver INFO Calculating throughput took [0.000473] seconds.
2018-07-03 07:32:23,391 PID:21120 rally.driver INFO Storing throughput took [0.000242] seconds.
2018-07-03 07:32:23,391 PID:21120 rally.driver INFO Flushing the metrics store took [0.000171] seconds.
2018-07-03 07:32:23,391 PID:21120 rally.driver INFO Postprocessing [1] raw samples took [0.001126] seconds in total.
2018-07-03 07:32:23,392 PID:21120 rally.metrics INFO Compression changed size of metric store from [96] bytes to [618] bytes
2018-07-03 07:32:23,393 PID:21120 rally.driver INFO Scheduling next task for client id [0] at their timestamp [8897255.744118] (master timestamp [8897255.746886])
2018-07-03 07:32:23,395 PID:21120 rally.driver INFO Scheduling next task for client id [1] at their timestamp [8897255.743212] (master timestamp [8897255.746886])
2018-07-03 07:32:23,395 PID:21113 rally.racecontrol INFO Task has finished.

Jessica254 · July 3, 2018, 8:17am

the 8th part:
2018-07-03 07:32:23,395 PID:21113 rally.racecontrol INFO Bulk adding request metrics to metrics store.
2018-07-03 07:32:23,396 PID:21113 rally.metrics INFO Restoring in-memory representation of metrics store.
2018-07-03 07:32:23,396 PID:21126 rally.driver INFO LoadGenerator[0] is continuing its work at task index [3] on [8897255.744118], that is in [0:00:00.994794].
2018-07-03 07:32:23,396 PID:21120 rally.driver INFO Scheduling next task for client id [2] at their timestamp [8897255.743247] (master timestamp [8897255.746886])
2018-07-03 07:32:23,397 PID:21127 rally.driver INFO LoadGenerator[1] is continuing its work at task index [3] on [8897255.743212], that is in [0:00:00.992307].
2018-07-03 07:32:23,398 PID:21120 rally.driver INFO Scheduling next task for client id [3] at their timestamp [8897255.741127] (master timestamp [8897255.746886])
2018-07-03 07:32:23,399 PID:21128 rally.driver INFO LoadGenerator[2] is continuing its work at task index [3] on [8897255.743247], that is in [0:00:00.990668].
2018-07-03 07:32:23,400 PID:21120 rally.driver INFO Scheduling next task for client id [4] at their timestamp [8897255.741825] (master timestamp [8897255.746886])
2018-07-03 07:32:23,400 PID:21129 rally.driver INFO LoadGenerator[3] is continuing its work at task index [3] on [8897255.741127], that is in [0:00:00.987024].
2018-07-03 07:32:23,401 PID:21120 rally.driver INFO Scheduling next task for client id [5] at their timestamp [8897255.741463] (master timestamp [8897255.746886])
2018-07-03 07:32:23,401 PID:21120 rally.driver INFO Scheduling next task for client id [6] at their timestamp [8897255.741175] (master timestamp [8897255.746886])
2018-07-03 07:32:23,401 PID:21130 rally.driver INFO LoadGenerator[4] is continuing its work at task index [3] on [8897255.741825], that is in [0:00:00.987015].
2018-07-03 07:32:23,402 PID:21120 rally.driver INFO Scheduling next task for client id [7] at their timestamp [8897255.740897] (master timestamp [8897255.746886])
2018-07-03 07:32:23,402 PID:21131 rally.driver INFO LoadGenerator[5] is continuing its work at task index [3] on [8897255.741463], that is in [0:00:00.986149].
2018-07-03 07:32:23,402 PID:21132 rally.driver INFO LoadGenerator[6] is continuing its work at task index [3] on [8897255.741175], that is in [0:00:00.985160].
2018-07-03 07:32:23,403 PID:21133 rally.driver INFO LoadGenerator[7] is continuing its work at task index [3] on [8897255.740897], that is in [0:00:00.984193].
2018-07-03 07:32:24,389 PID:21129 rally.driver INFO LoadGenerator[3] starts driving now.
2018-07-03 07:32:24,389 PID:21132 rally.driver INFO LoadGenerator[6] starts driving now.
2018-07-03 07:32:24,389 PID:21129 rally.driver INFO LoadGenerator[3] reached join point [JoinPoint(2)].
2018-07-03 07:32:24,389 PID:21132 rally.driver INFO LoadGenerator[6] reached join point [JoinPoint(2)].
2018-07-03 07:32:24,390 PID:21130 rally.driver INFO LoadGenerator[4] starts driving now.
2018-07-03 07:32:24,390 PID:21133 rally.driver INFO LoadGenerator[7] starts driving now.
2018-07-03 07:32:24,390 PID:21131 rally.driver INFO LoadGenerator[5] starts driving now.
2018-07-03 07:32:24,390 PID:21130 rally.driver INFO LoadGenerator[4] reached join point [JoinPoint(2)].
2018-07-03 07:32:24,390 PID:21133 rally.driver INFO LoadGenerator[7] reached join point [JoinPoint(2)].
2018-07-03 07:32:24,390 PID:21131 rally.driver INFO LoadGenerator[5] reached join point [JoinPoint(2)].
2018-07-03 07:32:24,391 PID:21127 rally.driver INFO LoadGenerator[1] starts driving now.
2018-07-03 07:32:24,391 PID:21127 rally.driver INFO LoadGenerator[1] reached join point [JoinPoint(2)].
2018-07-03 07:32:24,391 PID:21128 rally.driver INFO LoadGenerator[2] starts driving now.
2018-07-03 07:32:24,391 PID:21120 rally.driver INFO [1/8] drivers reached join point [2/5].
2018-07-03 07:32:24,391 PID:21128 rally.driver INFO LoadGenerator[2] reached join point [JoinPoint(2)].
2018-07-03 07:32:24,392 PID:21120 rally.driver INFO [2/8] drivers reached join point [2/5].
2018-07-03 07:32:24,392 PID:21120 rally.driver INFO [3/8] drivers reached join point [2/5].
2018-07-03 07:32:24,393 PID:21120 rally.driver INFO Resetting relative time of request metrics store.
2018-07-03 07:32:24,393 PID:21126 rally.driver INFO LoadGenerator[0] starts driving now.
2018-07-03 07:32:24,393 PID:21126 rally.driver INFO LoadGenerator[0] is executing [create-index].
2018-07-03 07:32:24,393 PID:21120 rally.driver INFO [4/8] drivers reached join point [2/5].
2018-07-03 07:32:24,393 PID:21126 rally.driver INFO Choosing [deterministic scheduler] for [create-index].
2018-07-03 07:32:24,393 PID:21120 rally.driver INFO [5/8] drivers reached join point [2/5].
2018-07-03 07:32:24,393 PID:21126 rally.driver INFO Creating iteration-count based schedule with [deterministic] distribution for [create-index] with [0] warmup iterations and [1] iterations.
2018-07-03 07:32:24,394 PID:21120 rally.driver INFO [6/8] drivers reached join point [2/5].
2018-07-03 07:32:24,394 PID:21120 rally.driver INFO [7/8] drivers reached join point [2/5].
2018-07-03 07:32:24,400 PID:21114 rally.mechanic INFO Resetting relative time of cluster system metrics store.
2018-07-03 07:32:24,403 PID:21119 rally.mechanic INFO Resetting relative time of system metrics store on host [[{'host': '10.3.68.178', 'port': 9200}]].
2018-07-03 07:32:24,428 PID:21126 elasticsearch WARNING PUT http://10.3.68.178:9200/geonames [status:400 request:0.033s]

Jessica254 · July 3, 2018, 8:17am

The last part:
2018-07-03 07:32:24,429 PID:21126 rally.driver ERROR Could not execute schedule
Traceback (most recent call last):
File "/usr/local/python3/lib/python3.6/site-packages/esrally/driver/driver.py", line 995, in call
total_ops, total_ops_unit, request_meta_data = execute_single(runner, self.es, params, self.abort_on_error)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/driver/driver.py", line 1065, in execute_single
raise exceptions.RallyAssertionError(msg)
esrally.exceptions.RallyAssertionError: ("Request returned an error. Error type: transport, Description: mapper_parsing_exception ({'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}], 'type': 'mapper_parsing_exception', 'reason': 'Failed to parse mapping [type]: No handler for type [text] declared on field [admin1_code]', 'caused_by': {'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}}, 'status': 400})", None)
2018-07-03 07:32:29,399 PID:21126 rally.driver INFO LoadGenerator[0] has detected a benchmark failure. Notifying master...
2018-07-03 07:32:29,401 PID:21120 rally.driver ERROR Main driver received a fatal exception from a load generator. Shutting down.
2018-07-03 07:32:29,401 PID:21120 rally.metrics INFO Closing metrics store.
2018-07-03 07:32:29,405 PID:21113 rally.racecontrol INFO Received a benchmark failure from [ActorAddr-(T|:59716)] and will forward it now.
2018-07-03 07:32:29,408 PID:21105 rally.racecontrol ERROR A benchmark failure has occurred
2018-07-03 07:32:29,408 PID:21105 rally.racecontrol INFO Telling benchmark actor to exit.
2018-07-03 07:32:29,410 PID:21113 rally.racecontrol INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring).
2018-07-03 07:32:29,411 PID:21105 rally.main INFO Attempting to shutdown internal actor system.
2018-07-03 07:32:29,412 PID:21114 rally.mechanic INFO MechanicActor#receiveMessage unrecognized(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:57423)])
2018-07-03 07:32:29,413 PID:21120 rally.driver INFO Main driver received ActorExitRequest and will terminate all load generators.
2018-07-03 07:32:29,414 PID:21113 rally.racecontrol INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring).
2018-07-03 07:32:29,414 PID:21119 rally.mechanic INFO Stopping nodes [hdh182, hdh178, hdh180] (due to ActorExitRequest)
2018-07-03 07:32:29,414 PID:21126 rally.driver INFO LoadGenerator[0] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,414 PID:21127 rally.driver INFO LoadGenerator[1] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,415 PID:21129 rally.driver INFO LoadGenerator[3] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,415 PID:21128 rally.driver INFO LoadGenerator[2] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,416 PID:21130 rally.driver INFO LoadGenerator[4] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,417 PID:21131 rally.driver INFO LoadGenerator[5] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,417 PID:21132 rally.driver INFO LoadGenerator[6] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,417 PID:21114 rally.mechanic ERROR Child actor exited with [ChildActorExited:ActorAddr-(T|:45117)] while in status [benchmark_started].
2018-07-03 07:32:29,420 PID:21133 rally.driver INFO LoadGenerator[7] is exiting due to ActorExitRequest.
2018-07-03 07:32:29,422 PID:21113 rally.racecontrol INFO BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:36997)] (ignoring).
2018-07-03 07:32:29,423 PID:21120 rally.driver INFO Load generator [0] has exited.
2018-07-03 07:32:29,423 PID:21120 rally.driver INFO Load generator [1] has exited.
2018-07-03 07:32:29,424 PID:21120 rally.driver INFO Load generator [3] has exited.
2018-07-03 07:32:29,424 PID:21120 rally.driver INFO Load generator [2] has exited.
2018-07-03 07:32:29,425 PID:21120 rally.driver INFO Load generator [4] has exited.
2018-07-03 07:32:29,425 PID:21120 rally.driver INFO Load generator [5] has exited.
2018-07-03 07:32:29,426 PID:21120 rally.driver INFO Load generator [6] has exited.
2018-07-03 07:32:29,427 PID:21120 rally.driver INFO Load generator [7] has exited.
2018-07-03 07:32:29,429 PID:21113 rally.racecontrol INFO BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:36997)] (ignoring).
2018-07-03 07:32:29,430 PID:21113 rally.racecontrol INFO BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:59716)] (ignoring).
2018-07-03 07:32:29,455 PID:21111 root INFO ---- Actor System shutdown
2018-07-03 07:32:29,457 PID:21105 rally.main INFO Actor system is still running. Waiting...
2018-07-03 07:32:30,458 PID:21105 rally.main INFO Shutdown completed.
2018-07-03 07:32:30,459 PID:21105 root ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 541, in dispatch_sub_command
race(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 471, in race
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 491, in with_actor_system
runnable(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 471, in
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 377, in run
raise e
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 374, in run
pipeline(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 63, in call
self.target(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 322, in benchmark_only
return race(cfg, external=True)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 276, in race
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: ('Error in load generator [0]', Request returned an error. Error type: transport, Description: mapper_parsing_exception ({'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}], 'type': 'mapper_parsing_exception', 'reason': 'Failed to parse mapping [type]: No handler for type [text] declared on field [admin1_code]', 'caused_by': {'type': 'mapper_parsing_exception', 'reason': 'No handler for type [text] declared on field [admin1_code]'}}, 'status': 400}))

danielmitterdorfer · July 3, 2018, 10:46am

Hi,

we see this in your track repository:

Jessica254:

[root@hdh154 ~/.rally/benchmarks/tracks/default]# git status
On branch master
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)
modified:   geonames/challenges/default.json
Untracked files:
(use "git add ..." to include in what will be committed)
geonames/__pycache__/

First of all, modifications here are problematic because they prevent switching branches. So I suggest you clear all local modifications with:

cd ~/.rally/benchmarks/tracks/default
git reset --hard master
git clean -xdf

This will clean all local changes in your repository. Secondly, I'm a bit puzzled about these log lines:

Rally detected that you will benchmark against a 2.x cluster but it checked out the "master" branch (and that is the reason the benchmark fails, because the mapping is not compatible with 2.x). It should have checkout out the branch "2" so I wonder whether this is due to your local modifications (see above on how to fix them) or that you do not have a clean clone locally available. If resetting the repository (see above) does not help, can you please run:

git branch -l

It should show:

  1
  2
  5
 master

Daniel

Jessica254 · July 3, 2018, 11:02am

I have cleaned the modification, and there's only branch master.

git reset --hard master
HEAD is now at e88b7fb Add tutorial track
[root@hdh154 ~/.rally/benchmarks/tracks/default]# git clean -xdf
Removing geonames/pycache/
[root@hdh154 ~/.rally/benchmarks/tracks/default]# git branch -l

master
[root@hdh154 ~/.rally/benchmarks/tracks/default]#

Topic		Replies	Views
Can't run race, Cannot find /root/.rally/benchmarks/data/geonames/documents-2.json.bz2 Elasticsearch rally	3	623	July 14, 2020
'Cannot find /home/node/.rally/benchmarks/data/http_logs/documents-191998.json.bz2. Please disable offline mode and retry again.' Elasticsearch rally	6	741	May 7, 2020
Esrally can't running 1.3.0 Elasticsearch rally	6	1812	October 10, 2019
Rally errors Elasticsearch rally	11	3224	July 5, 2017
The below bugs appers when running esrally Elasticsearch rally	29	7986	July 5, 2017

Esrally run failed for "Cannot find documents-2.json.bz2"

/ () ____ / / / /_____ ________ / /_ / / __ / __ `/ / __ / / __ / / _ / __/ / / / / / // / / / / // // / / / __/ // /// //_,// /__/_/_// ___/

[INFO] SUCCESS (took 257 seconds)

/ () ____ / / / /_____ ________ / /_ / / __ / __ `/ / __ / / __ / / _ / __/ / / / / / // / / / / // // / / / __/ // /// //_,// /__/_/_// ___/

[INFO] SUCCESS (took 152 seconds)

Related topics

/ () __ / / / /_ ______
/ /_ / / / `/ / / / / / _
/ / / / / / / // / / / / // // / / / /
// /// //_,// /__/_/_// ___/

/ () __ / / / /_ ______
/ /_ / / / `/ / / / / / _
/ / / / / / / // / / / / // // / / / /
// /// //_,// /__/_/_// ___/