# IO / Disc "tear down" for elastic search **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [September 9, 2020, 1:33pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013 "2020-09-09T13:33:41Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 15, 2020, 7:43pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/21 "2020-09-15T19:43:45Z")
But running on the same machine is kind of "Shoot in your own leg " 🙂 , right? I checked the network - between machines it is 4 Gb /s it is way, way more bandwidth that I generate. Also, is there a way I fix the exception? Again I am trying to follow the same command, running the same load but it is 20 times less then @dliappis has just generated. Something very basic is wrong but what exactly I don't know. I am very new to Rally. Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 15, 2020, 8:09pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/22 "2020-09-15T20:09:46Z")
If you are looking to simulate a production scenario and are looking for realism it is indeed a very bad thing to do. Given that this is a very artificial load and scenario anyway I do not see a problem with trying it and see what it gives. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [September 16, 2020, 8:02am UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/23 "2020-09-16T08:02:53Z")
> [@Oleg\_Ruchovets](#): > > Read the documentation at [Rally 1.4.1 — Rally 1.4.1 documentation](https://esrally.readthedocs.io/en/1.4.1/) You are running an old version of Rally. Can you switch to the latest (2.0.1)? Also, which version of Elasticsearch are you targeting? ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 16, 2020, 11:38am UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/24 "2020-09-16T11:38:24Z")
I installed Rally on the same host with ES. **user@oleg-elastic1:~/.rally/logs$ esrally --version** **esrally 2.0.1** **Elastic Search Version:** \*\* elasticsearch-7.8.1\*\* running command: esrally --track=pmc --track-params="bulk\_size:2000,bulk\_indexing\_clients:16" --target-hosts=localhost:9200 --pipeline=benchmark-only ``` 6.10 0.00 2.12 7.47 0.00 84.31 ``` Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r\_await w\_await aqu-sz rareq-sz wareq-sz svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 3775.00 0.00 42.52 0.00 4674.00 0.00 55.32 0.00 1.90 6.27 0.00 11.53 0.25 95.20 load statistics are the same: **iowait - 5.10** **MBs - 42** **w/s - 3500** Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/esrally/async\_connection.py", line 130, in perform\_request raw\_data = yield from response.text() File "/home/user/.local/lib/python3.8/site-packages/async\_timeout/ **init**.py", line 45, in **exit** self.\_do\_exit(exc\_type) File "/home/user/.local/lib/python3.8/site-packages/async\_timeout/ **init**.py", line 92, in \_do\_exit raise asyncio.TimeoutError asyncio.exceptions.TimeoutError 2020-09-16 11:16:25,190 -not-actor-/PID:26645 elasticsearch WARNING POST [http://localhost:9200/\_bulk](http://localhost:9200/_bulk) [status:N/A request:60.060s] Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/esrally/async\_connection.py", line 129, in perform\_request response = yield from self.session.request(method, url, data=body, headers=headers, timeout=request\_timeout) File "/home/user/.local/lib/python3.8/site-packages/aiohttp/client.py", line 504, in \_request await resp.start(conn) File "/home/user/.local/lib/python3.8/site-packages/aiohttp/client\_reqrep.py", line 847, in start message, payload = await self.\_protocol.read() # type: ignore # noqa File "/home/user/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 591, in read await self.\_waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/esrally/async\_connection.py", line 130, in perform\_request raw\_data = yield from response.text() File "/home/user/.local/lib/python3.8/site-packages/async\_timeout/ **init**.py", line 45, in **exit** self.\_do\_exit(exc\_type) File "/home/user/.local/lib/python3.8/site-packages/async\_timeout/ **init**.py", line 92, in \_do\_exit raise asyncio.TimeoutError asyncio.exceptions.TimeoutError 2020-09-16 11:16:30,101 -not-actor-/PID:26644 elasticsearch WARNING POST [http://localhost:9200/\_bulk](http://localhost:9200/_bulk) [status:N/A request:60.008s] Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/esrally/async\_connection.py", line 129, in perform\_request response = yield from self.session.request(method, url, data=body, headers=headers, timeout=request\_timeout) File "/home/user/.local/lib/python3.8/site-packages/aiohttp/client.py", line 504, in \_request await resp.start(conn) File "/home/user/.local/lib/python3.8/site-packages/aiohttp/client\_reqrep.py", line 847, in start message, payload = await self.\_protocol.read() # type: ignore # noqa File "/home/user/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 591, in read await self.\_waiter asyncio.exceptions.CancelledError running the command you send yesterday: esrally --pipeline=benchmark-only --track=eventdata --track-repository=eventdata --challenge=bulk-update --track-params=bulk\_size:10000,bulk\_indexing\_clients:16 --target-hosts=localhost:9200 --client-options="timeout:240" --kill-running-processes logs are without exceptions. performance is the same: avg-cpu: %user %nice %system %iowait %steal %idle 3.88 0.00 0.89 5.79 0.00 89.45 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r\_await w\_await aqu-sz rareq-sz wareq-sz svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 3804.00 0.00 41.15 0.00 5383.00 0.00 58.59 0.00 0.85 2.33 0.00 11.08 0.21 78.00 load statistics are the same: **iowait - 5.10** **MBs - 42** **w/s - 3500** how this can be? 🙂 Is there something I need to check? ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 16, 2020, 11:49am UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/25 "2020-09-16T11:49:18Z")
Also , Here is the resource consumption of the machine: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2b9e5e6a5176fa66bcdd8c2af1744587de941a47.png) ES ( java ) is loaded but rally process is almost not consuming resources. however, I see many esrally processes: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9e172b5d0463bbc4dfa0cb3b9bd0804810542cbf.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/3/73ec78d5e32d41ed9c3c100ac7e050b9926d58f7.png) What could be a problem? ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [September 16, 2020, 12:40pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/26 "2020-09-16T12:40:21Z")
> [@Oleg\_Ruchovets](#): > > File "/home/user/.local/lib/python3.8/site-packages/async\_timeout/ **init**.py", line 92, in \_do\_exit > raise asyncio.TimeoutError > asyncio.exceptions.TimeoutError > 2020-09-16 11:16:25,190 -not-actor-/PID:26645 elasticsearch WARNING POST [http://localhost:9200/\_bulk](http://localhost:9200/_bulk) [status:N/A request:60.060s] This tells us that at least one bulk request took longer than `60s` --\> Elasticsearch is struggling. Note you can increase the timeout via [client-options](https://esrally.readthedocs.io/en/stable/command_line_reference.html?highlight=timeout#id2), but you should probably lower the # of bulk clients; best leave both bulk\_size and # of indexing clients to default and start increasing little by little after you've had a good grasp of where your resources go. Rally having many processes and little CPU usage is good; you've specified 16 clients so at the very least you'll see the `min(num_of_cpus, 16)` as Rally processes. To get reasonable idea of your server resource usage I suggest the USE method and the commands mentioned: [Linux Performance Analysis in 60,000 Milliseconds | by Netflix Technology Blog | Netflix TechBlog](https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55) or alternatively install metricbeat: [Metricbeat quick start: installation and configuration | Metricbeat Reference [7.9] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/7.9/metricbeat-installation-configuration.html) When you are running `iostat`, are you certain you are running it against the disk that Elasticsearch writes to? I don't know if you have attached nvme/ssd scratch disks, is Elasticsearch writing to those? ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 16, 2020, 12:55pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/27 "2020-09-16T12:55:54Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/7/8/7814dc7bbd63c773b9832219981fcd7da6c3d5c0.png) sda is a disk coming with VM - this is where ES is installed ( it mostly idle) sdb is where data is written - it is attached drive. I am trying to cause this drive to "work hard" My problem is that I can't reach a fraction of what you are generating. 40MB/s and around 3500 write per second is very low thruput to reach disk saturation. and after plenty of different options, (even using multiple machines ) I still have the same load. What are the points to check to get a load similar to what you've got? (The only change I've done is OS paging cache is disabled, but when it was enabled it was 300 writes per second - which is nothing). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2020, 1:02pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/28 "2020-09-16T13:02:45Z")
Record disk utilization during an indexing run. In one of your data points above it was close to 100%. Elasticsearch does a lot of quite small random reads and writes so you are unlikely to reach throughput specified by the manufacturer, which are often measured using large sequential loads. ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 16, 2020, 1:15pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/29 "2020-09-16T13:15:34Z")
Ok , may you please share how to do it? Is there specific rally job / data set I need to run. Again I think I am doing wrong something very basic . I saw I am indexing a millions of documents or even 10s millions of documents. How this can be running the same command we've got 15 times differences? In general - I have only one machine and I try to "kill it with io load" and I can't get a fraction to its capacity. Sounds I am doing something wrong ? :-). I understand ( 32 CPU vs 16 but still we are running 16 rally processes and processes are doing nothing - resources utilized max 20 %) May you guys share more details where is the difference in configuration/version/setup, etc... or another way - how to generate the load similar and higher to what @dliappis shared ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2020, 1:19pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/30 "2020-09-16T13:19:15Z")
Look at the disk utilization while you are running, e.g. using iostat. Some of the data points indicated the storage is quite heavily utilised and possibly the bottleneck. ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 16, 2020, 1:24pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/31 "2020-09-16T13:24:02Z")
I am running **iostat** during all the tests I am doing these days. it didn't reach higher than 15% iowait on the peak and writes per second was 500. after I disable page caching become around 3000 writes per second. Again it is 40 Mb/s - which is very low for load testing and since a very basic smoke test cause for @dliappis to generate 600MB/s I want to understand how to get the same result or even higher. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2020, 1:25pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/32 "2020-09-16T13:25:14Z")
I asked about disk utilization, not iowait. ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 16, 2020, 1:28pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/33 "2020-09-16T13:28:00Z")
May you please point me which statistic on iostat you are referring to? [![](https://us1.discourse-cdn.com/elastic/original/3X/7/8/7814dc7bbd63c773b9832219981fcd7da6c3d5c0.png) ](https://us1.discourse-cdn.com/elastic/original/3X/7/8/7814dc7bbd63c773b9832219981fcd7da6c3d5c0.png) Where it was almost 100%? Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2020, 1:38pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/34 "2020-09-16T13:38:49Z")
The rightmost column in the iostat output, ideally tracked over time. ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 16, 2020, 2:42pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/35 "2020-09-16T14:42:35Z")
Ok, Thank you @Christian_Dahlqvist for pointing me on %utils, I will take a closer look at these statistics. But still, I want to get more load on the system. There are 70% - 80 % of free resources. How can I get a higher load? How can I reach close to what @dliappis had? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2020, 3:02pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/36 "2020-09-16T15:02:49Z")
I do not understand the purpose of this exercise. If you just want to see what throughput the disk can support I would recommend using fio which allows you to simulate different access patterns. Given that you are not simulating any realistic use case or scenario I do not understand why you are using Elasticsearch in this test. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [September 16, 2020, 3:09pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/37 "2020-09-16T15:09:57Z")
For the record `%util` in `iostat` is not a useful metric for flash memory type storage due to parallelism used in those devices; details [here](https://brooker.co.za/blog/2014/07/04/iostat-pct.html). It's a useful metric for rotational disks. Did you try the same benchmark on a different machine? Did you try the `index-and-query-logs-fixed-daily-volume` challenge using e.g. `esrally --track-repository=eventdata --track=eventdata --challenge=index-and-query-logs-fixed-daily-volume --track-params="number_of_days:10,daily_logging_volume:100GB" --pipeline=benchmark-only`? During execution you must record all load metrics (iostat -x, mpstat -P ANY, sar -n DEV,vmstat) to see if you are hitting machine bottlenecks. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [September 16, 2020, 3:10pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/38 "2020-09-16T15:10:36Z")
Also agreed with Christian that `fio` is a much better tool to test the maximum disk performance for different disk patterns. ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 17, 2020, 12:29pm UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/39 "2020-09-17T12:29:31Z")
Hello, Really appreciate detailed answers. running the command I've got the same exception mentioned before - can't execute it. I installed ES and Rally on the same machine (google cloud): c2-standard-60 (60 vCPUs, 240 GB memory), 3TB disk running command esrally --pipeline=benchmark-only --track=eventdata --track-repository=eventdata --challenge=bulk-update --track-params=bulk\_size:10000,bulk\_indexing\_clients:32 --target-hosts=localhost:9200 --client-options="timeout:240" --kill-running-processes left it to run for quite some time - 24 hours as you recommended. running around 1 hour: I do see writes around 400Mb from time to time using iostat but iowait% is very low. 2-3% and utils% is 30 - 40 % @Christian_Dahlqvist @dliappis Just wanted to explain the purpose of these tests: We are doing comparison different options for disks/storages for ES and my task is to reach the disk/io bottleneck. This task is not just to kill the disk but bring ES the limit of I/O and Disk. Hope this will make more sense why I am running these experiments Thanks ---
**Author:** ![Oleg\_Ruchovets](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleg_ruchovets/32/60137_2.png) [@Oleg\_Ruchovets](https://discuss.elastic.co/u/Oleg_Ruchovets)\ **Post date:** [September 18, 2020, 1:14am UTC](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013/40 "2020-09-18T01:14:37Z")
Hello , I am running the Rally for around 15 hours. esrally --pipeline=benchmark-only --track=eventdata --track-repository=eventdata --challenge=bulk-update --track-params=bulk\_size:10000,bulk\_indexing\_clients:32 --target-hosts=localhost:9200 --client-options="timeout:240" --kill-running-processes ES: ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/6/5649d415dd0e3ddce12852b88394c974a319839b.png) I do get a much higher load after sometime ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/d/bd67a1c8313874f1abd4f7fe95de6f93a504c4b5.jpeg) > [@dliappis](#): > > In the [reproduction scripts](https://gist.github.com/dliappis/54a9eedd5a272cf13b49ecaa85f87c7f) linked there you'll see we are relying on a [bulk-update](https://github.com/elastic/rally-eventdata-track/blob/e23b994e69b7b3ff8b0d386aa3b9a0a37f3e597d/eventdata/challenges/bulk-update.json#L28-L29) challenge; if you additionally set `id_seq_low_id_bias` to `true` and tweak the probability too you are going to end up with a very disk io heavy workload. I am running the bulk\_size challenge the first time as is but I do want to test it with the changes you recommended above. Is it possible to pass `id_seq_low_id_bias` to `true` and probability as a parameter. May you please give me an example of how to deal with probability tweaks and what is the logic behind them. Thanks in advance. [Previous page](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013.md?page=1) [Next page](https://discuss.elastic.co/t/io-disc-tear-down-for-elastic-search/248013.md?page=3)