I have a 6 node elastic cluster each with c5.4xlarge instances and writes to EBS volume with 5TB [10000 IOPS]. We only store 3 days worth of data and have a nightly job that terminates indexes older than 3 days.This has been working pretty good without any issues since an year indexing 1.5B documents a day to 3 different indexes.
Since we are using only 1.7-2TB out of the 5TB disk space i decided to create a new volume of 2.5TB [ 7500 IOPS] and attach to the servers and let the cluster take care of the balancing . I did this for couple of servers one server each day and once the cluster is balanced and all GREEN i did the other one. It's been a day since the cluster looks balanced and no errors but i started seeing lag in data being written to elastic at least by an hour now. Could this be because of reduction in the IOPS being provided? That is the only change that has been done.