What's eating node's memory?

[michaelr524 (Michael Ravits)]

Hi,

We are running a 3 node cluster, 25GB each, index size is 3TB at the moment.
Only running indexing, no searches have been made and heap usage moves between 75%-95%.
We are running two parallel indexers, each sending bulks of 1-10000 docs of a few bytes to a few kb each.

1. Are 3 nodes with 25GB each not enough for 3TB of data?
2. What is ES using all this memory for? I checked, and fielddata stands at 0 (no searches were made).
3. One of the nodes is always using 10% more heap than the rest, what could cause this?

ES version 1.7.3, previously were using 1.5.2 and getting the same behavior.

Thanks,
Michael

[nik9000 (Nik Everett)]

@michaelr524

We are running two parallel indexers, each sending bulks of 1-10000 docs of a few bytes to a few kb each.

10,000 docs is unlikely to squeeze into a few KB even if the docs are only 10 bytes each.

The real question is "are you seeing full GCs?" Elasticsearch should have a stat for that you can check next to the memory stats. If you aren't seeing many full GCs then you have nothing to worry about. Just ignore the memory usage - it'll get cleaned up eventually and it'll be done in the background.

@michaelr524

Are 3 nodes with 25GB each not enough for 3TB of data?

25 GB heaps? That should be fine.

How many indices? How many indices per node?

[michaelr524 (Michael Ravits)]

Hi Nik,

1 to 10k docs in each bulk where any of these docs could be a few bytes to a few kb each.

The reason I am asking is that after a few hours one of the nodes OOMs and becomes unresponsive to the rest of the cluster.

50 indices, 4 shards each. I think ES balances shards among nodes automatically, we have not done any special config changes for this.

Another detail which could matter is that all three nodes are running on the same physical machine. This server has 256 gb of RAM.

Thanks,
Michael

[Phani_Nadiminti]

Hi Michael,

How much heap memory you have allocated to each node on your cluster.you said each system is 25 GB of Memory allocated.

Thanks
phani

[michaelr524 (Michael Ravits)]

Hi Phani,

I have allocated 25GB of heap to each ES instance.
All three instances are running on the same physical machine (no VMs).

Thanks,
Michael

[michaelr524 (Michael Ravits)]

This seems like a similar problem:

Large heap usage with each node

So now i'm at at 352 shards over 44 indexes and I still see 12GB of heap usage per node. I'm thinking the only way I can really reach passed 1 billion docs on the 4 nodes is to add extra node per physical machine since I have the horse power., But keep 4 shards + 1 replica config. Btw i did some tests with my daily averages and I can even use single shard per index. But would that affect parallelism? I can also maybe provide yourkit snaphshot of what is eating up the ram...

[michaelr524 (Michael Ravits)]

Just checked, one node is non responsive, two others at 99% heap usage.

[michaelr524 (Michael Ravits)]

BTW, I inspected one of the core dumps a few days ago, when the cluster had 20gb heap per node. Out of 20gb, 13gb was used by byte arrays - does it make sense? Is there a way to find out what is in these 13gb?

[Phani_Nadiminti]

hi Michaelr,

I guess, Please check index level field data statics using following commands so that we can check which index is consuming more and which field in the index is consuming more memory.

GET /_stats/fielddata?fields=*

This link will be helpful : https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html

May be this will helpful to you.

Thanks,
phani

[michaelr524 (Michael Ravits)]

Hi,

We've been able to resolve the issue with some help from an ES professional.
Most of the memory was used by the segments area which must be the doc values, and there is not much which can be done in this regard but to add more nodes/memory.
We had some aggressive ngram mappings so we reduced/removed most of them and this reduced memory and disk usage significantly.

Thanks,
Michael