# Can't avoid swapping on Windows cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [January 22, 2017, 10:45pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414 "2017-01-22T22:45:03Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [January 22, 2017, 10:45pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/1 "2017-01-22T22:45:03Z")
We have a cluster that has three indexes (one main, and two supporting) with 5 shards and one replica. I am using 7 Windows nodes, each of which have 8 cores and 56gb RAM. I have set the heap to use 28gb. I have disabled the paging file in the OS and set mlock to true. No matter what I have tried, I am still seeing swap being used. ElasticHQ is reporting between 19 - 32 mb swap being used on the nodes. We have found that we have far exceeded the recommended shard size of 30 - 50gb. Our shards are currently 70 - 90 gb. We are gearing up to reindex and we will split out to more shards at that time. The swap is an issue we have had since we started, however (i.e. even when the shards will smaller). We are seeing some perf issues, and I can't help but be concerned about this metric that ElasticHQ is flagging in red. I would like to try to resolve this, but haven't had any luck yet. Thanks, ~john ---
**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Post date:** [January 23, 2017, 12:32am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/2 "2017-01-23T00:32:41Z")
Ignore the below. I thought you had not tried the options. This might be useful. [Disable Swapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [January 23, 2017, 2:59am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/3 "2017-01-23T02:59:24Z")
> [@jthoni](#): > > disabled the paging file in the OS and set mlock to true mlockall works only on Linux/Unix systems. I think disabling paging file should be enough. Personally, I don't trust the number on ElasticHQ for ES nodes running on Windows. ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [January 23, 2017, 3:06am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/4 "2017-01-23T03:06:28Z")
The info states that it is reporting: stat.os.swap.used\_in\_bytes / 1024 / 1024 It states that anything greater than 1 is a warning. It matches with what I see when querying in Sense, so I don't think HQ is misrepresenting anything ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [January 23, 2017, 12:32pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/5 "2017-01-23T12:32:16Z")
> [@anhlqn](#): > > mlockall works only on Linux/Unix systems. `bootstrap.memory_lock` in Elasticsearch 2.x and 5.x uses [`VirtualLock`](https://msdn.microsoft.com/en-us/library/windows/desktop/aa366895%28v=vs.85%29.aspx) on Windows to lock a specified region of the process's virtual address space into physical memory. What does Resource Monitor report for the process, specifically, commit and working set memory? Also, what does ```shell curl -XGET "http://localhost:9200/_nodes?filter_path=**.mlockall" ``` report? ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [January 23, 2017, 11:35pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/6 "2017-01-23T23:35:53Z")
I guess I mis interpreted this > The first option is to use **mlockall on Linux/Unix systems** , or VirtualLock on Windows, to try to lock the process address space into RAM, preventing any Elasticsearch memory from being swapped out. > [Disable swapping | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) Anyway, on my clusters I chose to disable paging file entirely, but ElasticHQ still reported bug number in swap. Is this `bootstrap.mlockall: true` still valid on 2.x? ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [January 24, 2017, 12:27am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/7 "2017-01-24T00:27:05Z")
- [Use `bootstrap.mlockall: true` in Elasticsearch 2.x \< 2.4.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/setup-configuration.html#setup-configuration-memory) - [Use `bootstrap.memory_lock: true` in Elasticsearch \>= 2.4.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-configuration.html#setup-configuration-memory) Take a look at [PR 18669](https://github.com/elastic/elasticsearch/pull/18669) for the change. It was renamed to remove the association between the Elasticsearch setting and `mlockall` 😄 what does ```auto curl -XGET "http://localhost:9200/_nodes?filter_path=**.mlockall" ``` return? I'm not sure where ElasticHQ is reporting swap from, but also take a look at that. ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [January 26, 2017, 10:52pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/8 "2017-01-26T22:52:16Z")
No, ElasticHQ does not misrepresent anything. It just reports the numbers it gets from `GET _nodes/stats`, `stats.os.swap.used_in_bytes` in this case. Before, I've tested both settings in ealier and 2.4.x as mentioned by @forloop > [@forloop](#): > > Use bootstrap.mlockall: true in Elasticsearch 2.x \< 2.4.0 > Use bootstrap.memory\_lock: true in Elasticsearch \>= 2.4.0 They never work. On my current ES 2.4.0 node, `bootstrap.memory_lock: true` is set, and paging file is disabled, yet the used swap is still a very big number `mlockall` is active ```auto "process" : { "refresh_interval_in_millis" : 1000, "id" : 7444, "mlockall" : true } ``` still high used swap in \_nodes/stats ```auto "swap": { "total_in_bytes": 207222677504, "free_in_bytes": 17371148288, "used_in_bytes": 189851529216 } ``` Physical memory is 192GB. Similar high swap is reported on all other Windows ES nodes ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [January 27, 2017, 2:09am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/9 "2017-01-27T02:09:26Z")
What does Resource Monitor report for the Java process, specifically, commit and working set memory? `mlockall: true` is reported only if the call [to set the process working set size via VirtualLock succeeds.](https://github.com/elastic/elasticsearch/blob/v2.4.0/core/src/main/java/org/elasticsearch/bootstrap/JNANatives.java#L130-L163) What do you see when monitoring the Page related counters under Performance Monitor: - `Page Faults/sec` - `Pages Input/sec` ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [January 27, 2017, 7:36pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/10 "2017-01-27T19:36:19Z")
This is stats for dedicated master node with 4GB heap size - Commit: 4GB - Working set: 4GB - Page faults/sec: 0 Avarage Pages Input/sec: 250 (total, not per process) ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [January 31, 2017, 12:23am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/11 "2017-01-31T00:23:19Z")
> [@anhlqn](#): > > This is stats for dedicated master node with 4GB heap size > > Commit: 4GB > Working set: 4GB > Page faults/sec: 0 > > Avarage Pages Input/sec: 250 (total, not per process) This looks like memory lock is set and working; commit and working set are at 4GB, with no page faults. What about on a data node? ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [February 2, 2017, 7:39pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/12 "2017-02-02T19:39:39Z")
Data node with 30GB heap size - Commit: 30GB - Working set: 40GB - Page faults/sec: 0 > [@forloop](#): > > This looks like memory lock is set and working; commit and working set are at 4GB, with no page faults. Does that mean ES is reporting the wrong swap usage number? ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [February 10, 2017, 5:05am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/13 "2017-02-10T05:05:25Z")
> [@anhlqn](#): > > Does that mean ES is reporting the wrong swap usage number? What does Resource Monitor show? ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [February 10, 2017, 9:22am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/14 "2017-02-10T09:22:56Z")
The numbers above were from Resource Monitor ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [February 13, 2017, 3:51am UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/15 "2017-02-13T03:51:11Z")
> [@anhlqn](#): > > Commit: 30GB > Working set: 40GB > Page faults/sec: 0 Commit is at 30GB, which matches the heap size specified, and page faults/sec is 0, which indicates to me that all of the virtual memory committed is physical memory i.e. RAM. Working set can be larger than commit as it includes both private and shareable bytes. If you'd like to understand more, you can run [VMMap](https://technet.microsoft.com/en-us/sysinternals/vmmap.aspx) on the process to get a more detailed breakdown. I hope that helps. ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [February 13, 2017, 5:24pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/16 "2017-02-13T17:24:26Z")
Thanks, I'll try VMMap. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 13, 2017, 5:25pm UTC](https://discuss.elastic.co/t/cant-avoid-swapping-on-windows-cluster/72414/17 "2017-03-13T17:25:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.