# Updating ELK causes triple write IOPS **URL:** \ **Category:** Elasticsearch\ **Created:** [May 29, 2018, 12:30pm UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681 "2018-05-29T12:30:20Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![JensVanDeynse](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)\ **Post date:** [May 29, 2018, 12:30pm UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/1 "2018-05-29T12:30:21Z")
Hello We (in our company) recently upgraded our ELK environment (using new servers) from 1.7.2 to 6.2.4. Another change we made is that before we used the logstash-forwarder on the hosts but now we use filebeat to forward the hosts. Next to this we also changed the cluster setup: - Previous setup - 1 server installed with Kibana, Elasticsearch & Logstash - New setup - 2 elasticsearch servers (node) - 1 server with kibana & logstash The amount of logs changed practically the same (a few more) but there is a lot more parsing (dividing our logs into more fields and mapping some fields to integer) but next to that only the version and the setup has changed. But we do notice that the old ELK stack (one server) has an average of 175 write IOPS while the new ELK stack has the following amount of write IOPS: - Master elasticsearch node: 542 write IOPS - Data elasticsearch node: 523 write IOPS Is there any known reason that this happens or is this a configuration mistake? Any help would be appreciated Thank you ---
**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\ **Post date:** [May 29, 2018, 3:12pm UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/2 "2018-05-29T15:12:58Z")
More fields of more types can definitely contribute to this. The storage configuration can also play a role. What is the storage setup on the old vs. the new system. In particular, does the new system use software RAID, especially 5 or 6? Also, what are your event rates? ---
**Author:** ![JensVanDeynse](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)\ **Post date:** [May 31, 2018, 6:26am UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/3 "2018-05-31T06:26:46Z")
Yes, I knew that more fields could be a reason for this but not in this order (the new cluster doesn't process three times as much logs so it seems weird that the write IOPS are so much higher). The storage configuration is exactly the same, no changes whatsoever The event rate by on the new ELK stack varies from 500 to 2500 (with peaks to 4000) events per second second, on the old one I can't really know exactly since we don't have any monitoring enabled on this but when I look at the indices the count of entries is the same (with a deviation of 5% max) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 31, 2018, 8:50am UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/4 "2018-05-31T08:50:16Z")
In Elasticsearch 6.x more data types use doc values than in Elasticsearch 1.x. This reduces heap usage but also results in additional disk I/O and larger size on disk. You may also want to compare mappings between the two versions to see if there are any other changes. ---
**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\ **Post date:** [May 31, 2018, 2:26pm UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/5 "2018-05-31T14:26:25Z")
You might also want to do a `GET /NAME-OF-INDEX/_settings` on a similar index running on both systems and compare the value of `refresh_interval` a new segment (which is basically a file) will be created at each refresh. A very short refresh interval could be also be contributing to more write IOPS. ---
**Author:** ![JensVanDeynse](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)\ **Post date:** [June 18, 2018, 6:52am UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/6 "2018-06-18T06:52:44Z")
The older version has already been taken offline but last week I changed this setting from 5s to 30s but this didn't really change anything for us. Thank you for your help ---
**Author:** ![JensVanDeynse](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@JensVanDeynse](https://discuss.elastic.co/u/JensVanDeynse)\ **Post date:** [June 18, 2018, 6:53am UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/7 "2018-06-18T06:53:35Z")
What do you mean with larger size on disk because with the new ELK cluster we use a lot less disk space for more data than before And what differences in Mapping could cause higher write IOPS? Thank you for your help ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2018, 7:00am UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/8 "2018-06-18T07:00:00Z")
Good point. There is more data going into doc values, which increase the size of those data types, but at the same time you no longer index a `_all` field, which will save a lot of space. If you use the best compression codec you could save even more space. One thing that has changed between 1.x and 6.x is that the transaction log is synced to disk much more frequently in order to improve resiliency, but I am not sure what impact this could have on the reported IOPS. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 16, 2018, 7:00am UTC](https://discuss.elastic.co/t/updating-elk-causes-triple-write-iops/133681/9 "2018-07-16T07:00:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.