Virtual memory is really only address space (and a lot of magic behind the scenes in the OS and hardware to make it work). It's not necessarily the case that at any moment in time the entire used address space of a process is mapped to physical memory. For Elasticsearch, the used virtual memory will increase as the heap grows, and as Lucene index segments are memory mapped. There's nothing to worry about, what you really care about from the perspective of physical memory is the resident set size.
Yes. You are right. We are not worried about virtual memory size.
We are concerned about the amount of physical memory consumed by elastic search, even with less data indexed in ES. I am using process working set size in windows task manager to measure the RAM consumption of the process.
Probably I know the cause of the issue.
Assuming the below configuration,
ES_HEAP_SIZE = 7GB
As we know JVM divides the heap info young and old generation and default size of young generation is 1/3 of heap. JVM young generation maximum size = ~ 2GB . JVM doesn't do GC till young generation is filled.
So even when minimal amount of data is indexed in elastic search (around 500MB) , still the RAM usage goes upto
2GB as GC is not done. And even after GC, JVM doesn't return the memory (RAM) to OS.
My question is can we tweak JVM parameters,
So it can reduce the RAM usage when elastic search has less data indexed.
I know we can reduce the ES_HEAP_SIZE for this, but my requirement is I always set
ES_HEAP_SIZE to half the machine memory, and elastic search should use more RAM only with
growing data indexed into it.
After JVM performs the GC, it should return the memory to OS.
Because I have a case when jvm heap size = ~200MB and process working set = ~2GB.