# What's eating node's memory? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 23, 2015, 10:14pm UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366 "2015-11-23T22:14:41Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![michaelr524](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@michaelr524](https://discuss.elastic.co/u/michaelr524)\ **Post date:** [November 23, 2015, 10:14pm UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/1 "2015-11-23T22:14:41Z")
Hi, We are running a 3 node cluster, 25GB each, index size is 3TB at the moment. Only running indexing, no searches have been made and heap usage moves between 75%-95%. We are running two parallel indexers, each sending bulks of 1-10000 docs of a few bytes to a few kb each. 1. Are 3 nodes with 25GB each not enough for 3TB of data? 2. What is ES using all this memory for? I checked, and fielddata stands at 0 (no searches were made). 3. One of the nodes is always using 10% more heap than the rest, what could cause this? ES version 1.7.3, previously were using 1.5.2 and getting the same behavior. Thanks, Michael ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 23, 2015, 10:31pm UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/2 "2015-11-23T22:31:09Z")
> [@michaelr524](#): > > We are running two parallel indexers, each sending bulks of 1-10000 docs of a few bytes to a few kb each. 10,000 docs is unlikely to squeeze into a few KB even if the docs are only 10 bytes each. The real question is "are you seeing full GCs?" Elasticsearch should have a stat for that you can check next to the memory stats. If you aren't seeing many full GCs then you have nothing to worry about. Just ignore the memory usage - it'll get cleaned up eventually and it'll be done in the background. > [@michaelr524](#): > > Are 3 nodes with 25GB each not enough for 3TB of data? 25 GB heaps? That should be fine. How many indices? How many indices per node? ---
**Author:** ![michaelr524](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@michaelr524](https://discuss.elastic.co/u/michaelr524)\ **Post date:** [November 23, 2015, 11:12pm UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/3 "2015-11-23T23:12:54Z")
Hi Nik, 1 to 10k docs in each bulk where any of these docs could be a few bytes to a few kb each. The reason I am asking is that after a few hours one of the nodes OOMs and becomes unresponsive to the rest of the cluster. 50 indices, 4 shards each. I think ES balances shards among nodes automatically, we have not done any special config changes for this. Another detail which could matter is that all three nodes are running on the same physical machine. This server has 256 gb of RAM. Thanks, Michael ---
**Author:** ![Phani\_Nadiminti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phani_nadiminti/32/45381_2.png) [@Phani\_Nadiminti](https://discuss.elastic.co/u/Phani_Nadiminti)\ **Post date:** [November 24, 2015, 5:36am UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/4 "2015-11-24T05:36:53Z")
Hi Michael, How much heap memory you have allocated to each node on your cluster.you said each system is 25 GB of Memory allocated. Thanks phani ---
**Author:** ![michaelr524](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@michaelr524](https://discuss.elastic.co/u/michaelr524)\ **Post date:** [November 24, 2015, 6:33am UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/5 "2015-11-24T06:33:24Z")
Hi Phani, I have allocated 25GB of heap to each ES instance. All three instances are running on the same physical machine (no VMs). Thanks, Michael ---
**Author:** ![michaelr524](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@michaelr524](https://discuss.elastic.co/u/michaelr524)\ **Post date:** [November 24, 2015, 6:53am UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/6 "2015-11-24T06:53:04Z")
This seems like a similar problem: > [@Large heap usage with each node](https://discuss.elastic.co/t/large-heap-usage-with-each-node/969/13): > > So now i'm at at 352 shards over 44 indexes and I still see 12GB of heap usage per node. I'm thinking the only way I can really reach passed 1 billion docs on the 4 nodes is to add extra node per physical machine since I have the horse power., But keep 4 shards + 1 replica config. Btw i did some tests with my daily averages and I can even use single shard per index. But would that affect parallelism? I can also maybe provide yourkit snaphshot of what is eating up the ram... ---
**Author:** ![michaelr524](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@michaelr524](https://discuss.elastic.co/u/michaelr524)\ **Post date:** [November 24, 2015, 4:28pm UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/7 "2015-11-24T16:28:35Z")
Just checked, one node is non responsive, two others at 99% heap usage. ---
**Author:** ![michaelr524](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@michaelr524](https://discuss.elastic.co/u/michaelr524)\ **Post date:** [November 24, 2015, 4:39pm UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/8 "2015-11-24T16:39:47Z")
BTW, I inspected one of the core dumps a few days ago, when the cluster had 20gb heap per node. Out of 20gb, 13gb was used by byte arrays - does it make sense? Is there a way to find out what is in these 13gb? ---
**Author:** ![Phani\_Nadiminti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phani_nadiminti/32/45381_2.png) [@Phani\_Nadiminti](https://discuss.elastic.co/u/Phani_Nadiminti)\ **Post date:** [November 30, 2015, 10:16am UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/9 "2015-11-30T10:16:10Z")
hi Michaelr, I guess, Please check index level field data statics using following commands so that we can check which index is consuming more and which field in the index is consuming more memory. GET /\_stats/fielddata?fields=\* This link will be helpful : [https://www.elastic.co/guide/en/elasticsearch/guide/current/\_limiting\_memory\_usage.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html) May be this will helpful to you. Thanks, phani ---
**Author:** ![michaelr524](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@michaelr524](https://discuss.elastic.co/u/michaelr524)\ **Post date:** [November 30, 2015, 10:52am UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/10 "2015-11-30T10:52:26Z")
Hi, We've been able to resolve the issue with some help from an ES professional. Most of the memory was used by the segments area which must be the doc values, and there is not much which can be done in this regard but to add more nodes/memory. We had some aggressive ngram mappings so we reduced/removed most of them and this reduced memory and disk usage significantly. Thanks, Michael ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:34pm UTC](https://discuss.elastic.co/t/whats-eating-nodes-memory/35366/11 "2017-07-05T23:34:54Z")