# Analyzing why ES Node has lots of i-o waits **URL:** \ **Category:** Elasticsearch\ **Created:** [July 3, 2013, 1:51pm UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645 "2013-07-03T13:51:51Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Eran\_Eidinger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_eidinger/32/44831_2.png) [@Eran\_Eidinger](https://discuss.elastic.co/u/Eran_Eidinger)\ **Post date:** [July 3, 2013, 1:51pm UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/1 "2013-07-03T13:51:51Z")
My cluster shows a lot of io-waits (about 50%). I do a lot of indexing and reindexing. I'm not terribly bothered with consistency, but more with "eventual" consistency. I can go as long as 5 minutes before data needs to be consistent. I thought maybe the re-indexing of lucene is the cause of much IO. Thought of maybe upping the refresh\_interval or maybe the index.translog options - is that the right way to go? My main problem is I do not know how to find out what my setting are. In [http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/) it lists alot of options, none of which are available when I use: curl -xget '[http://localhost:9200/my\_index/\_settings](http://localhost:9200/my_index/_settings)' I only get the number of shards, replicas. The elasticsearch.yml file does not tell what the defaults are. How would I know my changes took places, and what are the values now? Help much appreciated as I cant find documentation for this. ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [July 3, 2013, 2:15pm UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/2 "2013-07-03T14:15:47Z")
Hello, On Wed, Jul 3, 2013 at 4:51 PM, eranid [eranid@gmail.com](mailto:eranid@gmail.com) wrote: > My cluster shows a lot of io-waits (about 50%). > > I do a lot of indexing and reindexing. > I'm not terribly bothered with consistency, but more with "eventual" > consistency. I can go as long as 5 minutes before data needs to be > consistent. > I thought maybe the re-indexing of lucene is the cause of much IO. Thought > of maybe upping the refresh\_interval or maybe the index.translog options - > is that the right way to go? Should be. Another way to go is might be to increase indices.memory.index\_buffer\_size: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... Another way to go is to tweak merge policies (usually there's a trade-off between search performance and CPU+I/O load here): > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... To get a more clear idea of where your ES cluster is busy, I'd say you should monitor it with something like SPM: > **[Elasticsearch - Sematext Documentation](https://sematext.com/docs/integration/elasticsearch-integration/)** > > Collect and monitor key Elasticsearch metrics such as request latency, indexing rate, and segment merges with built-in anomaly detection, threshold, and heartbeat alerts. Send notifications to email and various chatops messaging services, correlate... > My main problem is I do not know how to find out what my setting are. In > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/) > it lists alot of options, none of which are available when I use: > > curl -xget '[http://localhost:9200/my\_index/\_settings](http://localhost:9200/my_index/_settings)' > > I only get the number of shards, replicas. The elasticsearch.yml file does > not tell what the defaults are. How would I know my changes took places, > and > what are the values now? You should see there the settings you changed by using the Update Settings API. If settings are not there, they should either be in the configuration file or ES is using the defaults. To find the defaults, you need to look in the documentation for each option. For example, indices.memory.index\_buffer\_size defaults to 10% (from the heap of the node). refresh\_interval defaults to 1s, and index.translog options are shown here: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 3, 2013, 4:56pm UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/3 "2013-07-03T16:56:53Z")
Can you clarify how you get this information? Is it a single node only with iowait? What is 50%? Is it constantly high or only peaks? Do you use Linux? iostat? How are your disks organized? If high iowait does not disappear and is constantly high, it may indicate a drive failure in a RAID. Jörg Am 03.07.13 15:51, schrieb eranid: > My cluster shows a lot of io-waits (about 50%). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Eran\_Eidinger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_eidinger/32/44831_2.png) [@Eran\_Eidinger](https://discuss.elastic.co/u/Eran_Eidinger)\ **Post date:** [July 3, 2013, 5:07pm UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/4 "2013-07-03T17:07:19Z")
Wow, thanks for the much detailed answer. Much obliged! ---
**Author:** ![Eran\_Eidinger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_eidinger/32/44831_2.png) [@Eran\_Eidinger](https://discuss.elastic.co/u/Eran_Eidinger)\ **Post date:** [July 3, 2013, 5:09pm UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/5 "2013-07-03T17:09:23Z")
@Jorg Hi, It is between 30-50% all the time (monitored by Newrelic). Single node on a m3.xl machine on AWS. It resides on an attached, non-root, EBS drive (non-EBS optimized). Eran. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 3, 2013, 5:46pm UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/6 "2013-07-03T17:46:07Z")
If you're on an EBS drive, there is not much you can do, maybe neighbors do saturate resources like network links. Jörg Am 03.07.13 19:09, schrieb eranid: > @Jorg > > Hi, > > It is between 30-50% all the time (monitored by Newrelic). Single node on a > m3.xl machine on AWS. > > It resides on an attached, non-root, EBS drive (non-EBS optimized). > > Eran. > > -- > View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Analyzing-why-ES-Node-has-lots-of-i-o-waits-tp4037475p4037493.html](http://elasticsearch-users.115913.n3.nabble.com/Analyzing-why-ES-Node-has-lots-of-i-o-waits-tp4037475p4037493.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [July 7, 2013, 3:20am UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/7 "2013-07-07T03:20:07Z")
Hey, not sure how newrelic gets that data itself... What do u see with iostat -x 1? Ebs may or may not be enough, depending on your case. Are you using PIOpS or std? We've had great improvements by tweaking how often commits to index happen ( anything from 5 to 120 secs depending on use case), and good old RAID . even RAID-0 on ephemeral disks is faster (and cheaper) than a single Ebs. For higher loads, the sky is the limit (50+ EBS raided vols can push a heck of a lot of iops..) On 04/07/2013 3:09 AM, "eranid" [eranid@gmail.com](mailto:eranid@gmail.com) wrote: > @Jorg > > Hi, > > It is between 30-50% all the time (monitored by Newrelic). Single node on a > m3.xl machine on AWS. > > It resides on an attached, non-root, EBS drive (non-EBS optimized). > > Eran. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Analyzing-why-ES-Node-has-lots-of-i-o-waits-tp4037475p4037493.html](http://elasticsearch-users.115913.n3.nabble.com/Analyzing-why-ES-Node-has-lots-of-i-o-waits-tp4037475p4037493.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:27am UTC](https://discuss.elastic.co/t/analyzing-why-es-node-has-lots-of-i-o-waits/12645/8 "2017-07-06T02:27:50Z")