# ES 5.6 gobbles up entire OS RAM after 30-40 mins of bulk indexing and no searching **URL:** \ **Category:** Elasticsearch\ **Created:** [May 27, 2019, 3:36pm UTC](https://discuss.elastic.co/t/es-5-6-gobbles-up-entire-os-ram-after-30-40-mins-of-bulk-indexing-and-no-searching/182940 "2019-05-27T15:36:32Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jiouser](https://avatars.discourse-cdn.com/v4/letter/j/e99b99/32.png) [@jiouser](https://discuss.elastic.co/u/jiouser)\ **Post date:** [May 27, 2019, 3:36pm UTC](https://discuss.elastic.co/t/es-5-6-gobbles-up-entire-os-ram-after-30-40-mins-of-bulk-indexing-and-no-searching/182940/1 "2019-05-27T15:36:32Z")
Dear All, We are using Elastic Search in production environment on 6 servers each of which have **80vcpus, 286GB of RAM** and with **2 ES 5.6 Data nodes** running on each of them. Each data node is given just less than **32GB of RAM** to ensure compressed **OOPS**. Spinning speed is set to HDD. Async flush is set. Translog size is set to 1GB. Refresh interval is set to 30s. mlock is true. swapiness is off. File descriptors count are above 2,00,000. Mmap count is 2612445. There is **one** **client** application which is doing bulk insertion in ES. This client application uses **16GB** of RAM residing on same server. The system is under bulk indexing load with **6 clients doing bulk insertion** in ES with each instance having batch size of **7500 with 2 concurrent actions**. The system is used not very often for searching(like 5-10 aggregation query once every hour at hour rollover). For **30-40** mins during load, the ES works properly (i.e. the nodes are taking in huge loads and working smoothly). **_Top command and free -g shows that system is healthy and my client is also able to out load_.** **After 40-45 mins** , the **free -g** on each of the server shows **0** under free and entire available memory under shared/cache. Now I do understand that I can still run applications as these are shared memory and available to any application which demands it, BUT my client starts experiencing lag due to no available memory and fails to work at same pace which leads to decreased throughput. I have gone through numerous post surrounding this. **None helped**. HOW do I **restrict ES/Lucene** to take up only **x%** of total **OS RAM** available on my OS. For example in above scenario, 2 ES Data Nodes - 64 GB. 1 client application - 16 GB Leaving equal amount of ES RAM to Lucene for caching - 64GB I should still be left with roughly 140GB of Memory by limting Lucene/ES to use only 64GB of my OS RAM. I have not found any variable, any parameter which can be set to restrict this. On some post I saw it might be due to netty or due to some bug, but I am not able to find any variable or parameter which can be tuned for this. Other applications(1 or 2) which run on same server as that of ES starts experiencing heavy lag and soon leads to decreased throughput. Due to limitation, I need to have atleast 1 or 2 instances of my application running on these servers. If any further information is needed, please do let me know. Please help. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 27, 2019, 6:28pm UTC](https://discuss.elastic.co/t/es-5-6-gobbles-up-entire-os-ram-after-30-40-mins-of-bulk-indexing-and-no-searching/182940/2 "2019-05-27T18:28:29Z")
> [@jiouser](#): > > HOW do I **restrict ES/Lucene** to take up only **x%** of total **OS RAM** available on my OS. For example in above scenario, As you point out this is shared/cached memory and this is an OS level setting that is outside Elasticsearch's ability to maintain. You'll have to look up memory management for your OS. ---
**Author:** ![jiouser](https://avatars.discourse-cdn.com/v4/letter/j/e99b99/32.png) [@jiouser](https://discuss.elastic.co/u/jiouser)\ **Post date:** [May 28, 2019, 9:27am UTC](https://discuss.elastic.co/t/es-5-6-gobbles-up-entire-os-ram-after-30-40-mins-of-bulk-indexing-and-no-searching/182940/3 "2019-05-28T09:27:26Z")
HI Mark, This is not OS level issue. Every time bulk load is running for 30-40 mins, the entire cache is taken up by ES process. This makes all the other applications running on same server to respond slowly. I tried changing file store to niofs as well but to no avail. The RAM used by ES still reaches 100%. ES alone takes up 100% of RAM in cache/virtual memory. There has to be some way by which I can restrict ES to occupy some % of shared/virtual memory. ES version is 5.6 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2019, 8:59pm UTC](https://discuss.elastic.co/t/es-5-6-gobbles-up-entire-os-ram-after-30-40-mins-of-bulk-indexing-and-no-searching/182940/4 "2019-05-28T20:59:12Z")
> [@jiouser](#): > > There has to be some way by which I can restrict ES to occupy some % of shared/virtual memory. As I mentioned there is, it's via the OS. ---
**Author:** ![jiouser](https://avatars.discourse-cdn.com/v4/letter/j/e99b99/32.png) [@jiouser](https://discuss.elastic.co/u/jiouser)\ **Post date:** [May 29, 2019, 4:25pm UTC](https://discuss.elastic.co/t/es-5-6-gobbles-up-entire-os-ram-after-30-40-mins-of-bulk-indexing-and-no-searching/182940/5 "2019-05-29T16:25:59Z")
Hi Mark. Thanks for the reply. What I have been trying to say is when I give ES 32 G of Heap and leave rest for Lucene to do its magic while my other applications on same server are running, Lucene+ES (in **top** command on RHEL, the user is still **es** using which ES was booted up), es user takes up **entire virtual memory** leading to **complete starvation** of my other application running on same physical hardware. I have gone through numerous post to no avail. If you can assist me in identifying **parameters** in **OS** which can be changed to restrict this, it would be really helpful. Huge pages are **disabled**. I tried **limiting** virtual memory in limits.conf however ES did not even start due to bootstrap checks for unlimited virtual memory. I tried putting in Java opts, **NoPreferDirect** but even that did not helped. Tried by **disabling ExplicitGC** did not work. Tried changing the **netty to netty4** , but ES gave an exception that it has to be security3 or security4 and neither of them worked eventually. Changed file to " **niofs**". Did not worked. Did other changes in OS to no avail. Please guide me on OS level setting if it can be controlled via OS. We are completely lost in finding a solution for this. I am running **ES 5.6 on RHEL 7.5** ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 26, 2019, 4:26pm UTC](https://discuss.elastic.co/t/es-5-6-gobbles-up-entire-os-ram-after-30-40-mins-of-bulk-indexing-and-no-searching/182940/6 "2019-06-26T16:26:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.