# Increasing bandwidth for Rally results

**URL:** <https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [September 30, 2020, 1:55am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434 "2020-09-30T01:55:15Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [September 30, 2020, 1:55am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/1 "2020-09-30T01:55:15Z")

</div>

Hi,  
I have my Rally instance on a different host ( not part of my ECK 1.2 ). When I run http\_logs using below command '''esrally --pipeline benchmark-only --track=http\_logs --challenge append-no-conflicts-index-only --target-host=IP:9200 --client-options="use\_ssl:true,basic\_auth\_user:'elastic',basic\_auth\_password:'password',verify\_certs:false'''

I get a throughput of 160K ops/s for index-append.

How can I increase this throughput as my ES search cluster is not bottlenecked of N/W, CPU or memory ?

Is it possible to have multiple Elastic Search End Points or any other suggestion , please let me know.

Thanks  
Kailas

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 30, 2020, 6:55am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/2 "2020-09-30T06:55:11Z")

</div>

Indexing is very I/O intensive, so the bottleneck is often storage performance rather than the resources you have listed. Are you monitoring disk I/O and utilization?

---

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [October 6, 2020, 7:01am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/3 "2020-10-06T07:01:18Z")

</div>

Yes, diskIO is hardly 140 MB/s ( have 25 G pipe) and utilization is not even 30%.

Is it possible to have multiple end points of elastic search and then I load balance the same ?

Please let me know your thoughts.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 6, 2020, 7:11am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/4 "2020-10-06T07:11:00Z")

</div>

What does iowait look like?

---

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [October 6, 2020, 9:55am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/5 "2020-10-06T09:55:56Z")

</div>

On Rally VM  
[root@rallyvm .rally]# iostat  
Linux 3.10.0-1062.el7.x86\_64 (rallyvm) 10/06/2020 _x86\_64_ (32 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle  
0.24 0.00 0.02 0.00 0.00 99.74

Device: tps kB\_read/s kB\_wrtn/s kB\_read kB\_wrtn  
sda 0.03 0.14 0.31 131248 294067  
sdb 1.06 364.62 121.87 350493978 117144317  
dm-0 1.09 364.71 122.11 350583605 117377180  
dm-1 0.02 0.01 0.06 6784 53476  
dm-2 0.00 0.00 0.01 1288 5660

On K8 master  
[root@k8smaster elastic]# iostat  
Linux 3.10.0-1062.el7.x86\_64 (k8smaster) 10/06/2020 _x86\_64_ (8 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle  
0.59 0.00 0.33 0.02 0.00 99.06

Device: tps kB\_read/s kB\_wrtn/s kB\_read kB\_wrtn  
sda 9.07 0.43 48.32 2387038 268057432  
dm-0 9.10 0.42 48.32 2352914 268042452  
dm-1 0.00 0.00 0.00 2072 0  
dm-2 0.00 0.00 0.00 1974 12087

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 6, 2020, 10:13am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/6 "2020-10-06T10:13:06Z")

</div>

Was that taken on the data node during peak indexing? 99% idle CPU and 160k EPS sounds odd.

---

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [October 6, 2020, 10:42am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/7 "2020-10-06T10:42:18Z")

</div>

ok, I understand. I took it on Rally and K8s master node.

> Blockquote

One of the data nodes:  
[root@node4 ~]# iostat  
Linux 3.10.0-1127.el7.x86\_64 (node4) 10/06/2020 _x86\_64_ (80 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle  
0.63 0.00 0.44 0.07 0.00 98.86

Device: tps kB\_read/s kB\_wrtn/s kB\_read kB\_wrtn  
sda 7.65 165.04 8.47 913433409 46882392  
sdb 36.18 2433.80 1398.16 13469807660 7738065457  
sdc 35.92 2253.40 1389.84 12471381736 7692050625  
sdd 36.58 2282.68 1394.23 12633427300 7716308155  
sde 36.36 2390.10 1389.20 13227925260 7688465857  
sdf 36.04 2253.67 1390.06 12472853728 7693233753  
sdg 36.21 2280.55 1390.31 12621651148 7694614918  
sdh 35.72 2229.24 1393.40 12337670096 7711732821  
sdi 36.15 2145.21 1392.95 11872580444 7709245941  
sdj 35.69 2212.20 1385.91 12243354692 7670287379  
sdk 35.08 2067.33 1376.82 11441559468 7619941448  
dm-0 7.57 165.03 8.47 913356664 46878260  
dm-1 0.00 0.00 0.00 2072 0  
dm-2 0.00 0.00 0.00 1328 2048  
scinia 2.75 0.00 221.30 2610 1224758020

> Blockquote

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [October 6, 2020, 11:12am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/8 "2020-10-06T11:12:05Z")

</div>

Note that `iostat` almost always provides very little value without extended metrics via the `-x` flag. Try using `iostat -xd` instead.

I also have another ask:

Please don't post unformatted code as it's very hard to read.

Instead, paste the text and format it with \</\> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your posts to solve this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 3, 2020, 11:12am UTC](https://discuss.elastic.co/t/increasing-bandwidth-for-rally-results/250434/9 "2020-11-03T11:12:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
