Why the JVM heap of ES is always about 50%?


(Keith Tt) #1

ES version: 5.5.1

Is this a coincidence or something else?


(Mark Walkom) #2

Because we leverage the OSs ability to cache commonly accessed files in the remainder of the memory.


(Keith Tt) #3

So... how do I decide the heap size of ES? 2gb by default is ok?


(Mark Walkom) #4

That's a massive - it depends.

What sort of data, how much, what sort of queries, what version, what JVM, what SLAs do you have?


(Keith Tt) #5

The data is nginx access log, about 100gb every day, java is open-jdk-1.8.0.

I use ELK to collect and analysis nginx logs, how to get the info you mentioned above...?


(Timo Witte) #6

We generally use 3/4 of the physically available RAM as the JVM HeapSize for Elasticsearch..
More RAM = Better as Lucene can keep more segments in RAM.

But depends on the queries.. If you create indeces per day and only access the last few days you wan´t need as much RAM as if you access all the data all the time.. Ideally you want to be able to keep all interactively browsed data in memory.

As much as i love Free Software, don´t use open-jdk it´s much slower than the properitary OracleJVM..


(Keith Tt) #7

Really?:flushed:

It's very slow when I view the dashboard in 7 days...


(Aaron Mildenstein) #8

The out of the box settings for Elasticsearch do not have the JVM triggering a garbage collection until the heap reaches 70%. The goal for most users of Elasticsearch should be to keep the heap at or near 50% after a GC has completed. If the heap is near, or above 70% on a regular or continual basis (or cannot clean up RAM after a GC so it stays around 70%), that's evidence of memory pressure, which indicates a need for more heap, or more nodes, or perhaps both.


(Keith Tt) #9

Great!!! Thanks a lot for your explanation, remove many my doubts! :grinning:


(Keith Tt) #10

@warkolm @Spacefish @theuntergeek

By the way, what about the heap size of logstash?


(Mark Walkom) #11

What about it exactly would you like to know?


(Keith Tt) #12

Maybe it should be percent of physical memory?


(Mark Walkom) #13

Not really, it only needs to hold a small amount of data in memory, so unless you have very large events, a few GB should be sufficient.


(Keith Tt) #14

How to check the size of events?


(Ravi Shanker Reddy) #15

Depends on so many cases.

A Basic one: A single line in your log file is an event. The size of the line becomes the size of the event.

It varies if you are creating new variables, or using multi line and so many other ways


(Keith Tt) #16

Thanks, I found Events Received and Events Emitted on monitoring of kibana...


(Keith Tt) #17

Replace openjdk by oracleJDK, load average decrease much, amazing...Thanks a lot!!!:kissing_heart:


(system) #18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.