Running on Elastic Search data load using Rally and it is failing 80/20 all the time.
esrally --track=contactdata --target-hosts=esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200 --track-params="bulk_size:2000" --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:''"
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Writing logs to /root/.rally/logs/rally_out_20180418T190653Z.log
************************************************************************
************** WARNING: A dark dungeon lies ahead of you **************
************************************************************************
Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.
Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).
************************************************************************
****** Use this pipeline only if you are aware of the tradeoffs. ******
*************************** Watch your step! ***************************
************************************************************************
[INFO] Racing on track [contactdata], challenge [bulk-index-throughput-test] and car ['external'] with version [6.2.2].
[WARNING] flush_total_time is 9256 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 3048348 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_throttled_time is 212097 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 3105643 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_time is 6453799 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index [100% done]
Running create-index [100% done]
Running cluster-health [100% done]
Running index-append [100% done]
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Lap | Metric | Task | Value | Unit |
|------:|------------------------------:|-------------:|---------:|-------:|
| All | Indexing time | | 50.8068 | min |
| All | Indexing throttle time | | 0 | min |
| All | Merge time | | 107.565 | min |
| All | Refresh time | | 51.764 | min |
| All | Flush time | | 0.154267 | min |
| All | Merge throttle time | | 3.53495 | min |
| All | Total Young Gen GC | | 0.217 | s |
| All | Total Old Gen GC | | 0 | s |
| All | Heap used for segments | | 25.8756 | MB |
| All | Heap used for doc values | | 1.88224 | MB |
| All | Heap used for terms | | 21.436 | MB |
| All | Heap used for norms | | 0.234131 | MB |
| All | Heap used for points | | 0.908391 | MB |
| All | Heap used for stored fields | | 1.41486 | MB |
| All | Segment count | | 395 | |
| All | Min Throughput | index-append | 27.55 | docs/s |
| All | Median Throughput | index-append | 27.55 | docs/s |
| All | Max Throughput | index-append | 27.55 | docs/s |
| All | 50th percentile latency | index-append | 247.041 | ms |
| All | 100th percentile latency | index-append | 251.092 | ms |
| All | 50th percentile service time | index-append | 247.041 | ms |
| All | 100th percentile service time | index-append | 251.092 | ms |
| All | error rate | index-append | 20 | % |
--------------------------------
[INFO] SUCCESS (took 26 seconds)
(EDIT: added code block by wrapping in ``` )