Increasing bandwidth for Rally results

Elastic Stack Elasticsearch
1

Hi,
I have my Rally instance on a different host ( not part of my ECK 1.2 ). When I run http_logs using below command '''esrally --pipeline benchmark-only --track=http_logs --challenge append-no-conflicts-index-only --target-host=IP:9200 --client-options="use_ssl:true,basic_auth_user:'elastic',basic_auth_password:'password',verify_certs:false'''

I get a throughput of 160K ops/s for index-append.

How can I increase this throughput as my ES search cluster is not bottlenecked of N/W, CPU or memory ?

Is it possible to have multiple Elastic Search End Points or any other suggestion , please let me know.

Thanks
Kailas

2

Indexing is very I/O intensive, so the bottleneck is often storage performance rather than the resources you have listed. Are you monitoring disk I/O and utilization?

3

Yes, diskIO is hardly 140 MB/s ( have 25 G pipe) and utilization is not even 30%.

Is it possible to have multiple end points of elastic search and then I load balance the same ?

Please let me know your thoughts.

4

What does iowait look like?

5

On Rally VM
[root@rallyvm .rally]# iostat
Linux 3.10.0-1062.el7.x86_64 (rallyvm) 10/06/2020 x86_64 (32 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
0.24 0.00 0.02 0.00 0.00 99.74

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.03 0.14 0.31 131248 294067
sdb 1.06 364.62 121.87 350493978 117144317
dm-0 1.09 364.71 122.11 350583605 117377180
dm-1 0.02 0.01 0.06 6784 53476
dm-2 0.00 0.00 0.01 1288 5660

On K8 master
[root@k8smaster elastic]# iostat
Linux 3.10.0-1062.el7.x86_64 (k8smaster) 10/06/2020 x86_64 (8 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
0.59 0.00 0.33 0.02 0.00 99.06

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 9.07 0.43 48.32 2387038 268057432
dm-0 9.10 0.42 48.32 2352914 268042452
dm-1 0.00 0.00 0.00 2072 0
dm-2 0.00 0.00 0.00 1974 12087

6

Was that taken on the data node during peak indexing? 99% idle CPU and 160k EPS sounds odd.

7

ok, I understand. I took it on Rally and K8s master node.

Blockquote

One of the data nodes:
[root@node4 ~]# iostat
Linux 3.10.0-1127.el7.x86_64 (node4) 10/06/2020 x86_64 (80 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
0.63 0.00 0.44 0.07 0.00 98.86

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 7.65 165.04 8.47 913433409 46882392
sdb 36.18 2433.80 1398.16 13469807660 7738065457
sdc 35.92 2253.40 1389.84 12471381736 7692050625
sdd 36.58 2282.68 1394.23 12633427300 7716308155
sde 36.36 2390.10 1389.20 13227925260 7688465857
sdf 36.04 2253.67 1390.06 12472853728 7693233753
sdg 36.21 2280.55 1390.31 12621651148 7694614918
sdh 35.72 2229.24 1393.40 12337670096 7711732821
sdi 36.15 2145.21 1392.95 11872580444 7709245941
sdj 35.69 2212.20 1385.91 12243354692 7670287379
sdk 35.08 2067.33 1376.82 11441559468 7619941448
dm-0 7.57 165.03 8.47 913356664 46878260
dm-1 0.00 0.00 0.00 2072 0
dm-2 0.00 0.00 0.00 1328 2048
scinia 2.75 0.00 221.30 2610 1224758020

Blockquote

8

Note that iostat almost always provides very little value without extended metrics via the -x flag. Try using iostat -xd instead.

I also have another ask:

Please don't post unformatted code as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your posts to solve this.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.