# I/O wait is increasing after many reindexing **URL:** \ **Category:** Elasticsearch\ **Tags:** docker, reindex\ **Created:** [July 5, 2023, 3:21pm UTC](https://discuss.elastic.co/t/i-o-wait-is-increasing-after-many-reindexing/337695 "2023-07-05T15:21:56Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![yannlef](https://avatars.discourse-cdn.com/v4/letter/y/4491bb/32.png) [@yannlef](https://discuss.elastic.co/u/yannlef)\ **Post date:** [July 5, 2023, 3:21pm UTC](https://discuss.elastic.co/t/i-o-wait-is-increasing-after-many-reindexing/337695/1 "2023-07-05T15:21:56Z")
Hello, I'm trying to find some response here, since I think I exhausted all my solutions on that problem. I'm also totally new with this subject. I am working on an old ELK (5.5), hosted on a docker in a dedicated VM hosted on mutualised server. A list of actions had been made on it that had the consequence to dramatically increase the I/O Wait. 1. The VM has been put on a different disk, better faster and so on.. 2. It has been restarted 3. Since it didn't worked properly, they restarted also the docker inside 4. During two weeks, a colleague made each day a reindexing on the whole ELK -\> what you can see during this period is that sometimes the I/O wait increases and some times it gets back to its old values. As you can see on the stats on the attached image, sometimes the I/O wait increases and sometimes it gets back to normal during that period. And now the I/O wait has increased so dramatically that the ELK slows down so much that some times the whole internet site is freezing. We have 5 nodes on the cluster, the swap seems to be off and there is not much to read on the ealsticsearch.yml So far, we tried force-merge and reindexing+forcemerge.. that seems to have increased the problem May anyone experienced something similar and may have an idea about the causes and the solutions? The stats for that period : ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/5/658640bbff1ccec7e421d3f8159e880279e6a388.jpeg) I stay at disposal to answer any question Thank you very much for your help ---
**Author:** ![yannlef](https://avatars.discourse-cdn.com/v4/letter/y/4491bb/32.png) [@yannlef](https://discuss.elastic.co/u/yannlef)\ **Post date:** [July 6, 2023, 2:55pm UTC](https://discuss.elastic.co/t/i-o-wait-is-increasing-after-many-reindexing/337695/2 "2023-07-06T14:55:05Z")
Hello, i have some more information. Apparently, those redindexing were coupled with force merge. So apparently we have one very big segment and few very little, and that is what makes this important i/o wait ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [July 6, 2023, 11:19pm UTC](https://discuss.elastic.co/t/i-o-wait-is-increasing-after-many-reindexing/337695/3 "2023-07-06T23:19:46Z")
I'm surprised the bot hasn't come by here yet, but Elasticsearch 5.x is far past End of Life and you should really look into upgrading as a matter of urgency. You will not find many people around here with 5.x experience, or those who have it probably haven't used it in a long time. (I _do not_ have any 5.x experience, so take what I'm about to say with a grain of salt) _Any_ sort of I/O wait means that the server is waiting on disks for work, this indicates that Elasticsearch is trying to do more work than the disks can keep up with. With this piece of information in mind there are a few things to point out: 1. Both re-index _and_ force-merge are I/O intensive functions. Seeing I/O utilization spike during these is to be expected. - Whether you see I/O wait increase goes back to the disks being the limiting factor here. 2. Restarting an Elasticsearch node can be an I/O intensive function. Elasticsearch needs to do various things on startup that involve reading data. - I _think_ this has been improved in recent versions (again, extremely important to upgrade) Some additional things: - > the swap seems to be off - "Seems to be off", and "actually being off" are 2 different things, I'd confirm that swap is _actually_ off. - > The VM has been put on a different disk, better faster and so on. - What does this mean? - Is this local storage, or storage over some sort of network (SAN, NAS, etc...)? - What type of storage (spinning disk (HDD), SATA SSD, NVMe SSD, etc...)? - Can you provide more details? - It seems like you're running on Linux, what does `iostat` look like, what is the server waiting on? - How much data are we dealing with in this cluster? Overall, just from looking at this, it seems like your disks/storage are not able to keep up with the demands of Elasticsearch. (Again, there are probably improvements in newer versions and you should really be upgrading, but I'm not 100% sure they'll fix your problem.) ---
**Author:** ![yannlef](https://avatars.discourse-cdn.com/v4/letter/y/4491bb/32.png) [@yannlef](https://discuss.elastic.co/u/yannlef)\ **Post date:** [July 18, 2023, 8:21am UTC](https://discuss.elastic.co/t/i-o-wait-is-increasing-after-many-reindexing/337695/4 "2023-07-18T08:21:13Z")
Hello BenB196, Thank you for your reply. To answer your questions : - We couldn't change version because the application is a very old legacy still on Hibernates 5 and apparently it was to expensive to change (Im in charge but alone and there only since March, when the problem actually appeared). - The Swap IS of, I could see it in the docker properties - The Virtual Machine is now on SSD (don't know all the details) and he storage is not local, but SAN - We are dealing with files, something like 4billion, each of it has several information indexed in the ELK. It is continuously rewritten and read, day and night. It has simultaneously several massive import and updates through a BUS. - The nodes have 200Gb each and are using 50Gb for the storage Since last time, we tried some other things : - Deleting and recreating the index with reindexation, it did nothing - ForceMerge on 1 segment.. nothing - Deactivating services (less inputs).. nothing ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 15, 2023, 8:21am UTC](https://discuss.elastic.co/t/i-o-wait-is-increasing-after-many-reindexing/337695/5 "2023-08-15T08:21:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.