Logstash process taking more memory than configured

Elastic Stack Logstash
1

Hi there. Question in regards to some weird behavior I am noticing on all of our logstash servers. We are running 7.13.4 currently. The problem is related to logstash seemingly taking more memory than I have configured in the jvm.options file. I am getting this information via top. We have the process configured to use just over 50% of the total ram available on the machine as it is the only purpose of the server. After, I bounce the process the memory consumed or held by the logstash process according to top will get as high as 95% (very slowly) until I reboot the process. Wondering if anyone else has seen this or has any ideas.

Thanks,
Alex

2

The memory configured in jvm.options is only the JVM heap. A JVM may require hundreds of megabytes of additional memory above that. The pmap command should give you an idea of what memory is being used. If you run that twice, once when the JVM has started and once when it has grown it should give you an idea of where the memory is being used.

I have seen J2EE JVMs leak tens of gigabytes of native (not heap) memory which we believed was due to finalization delays. I am sure there are other possible leaks too.

3

It can also be due to the OS caching files that Logstash is reading, which will be associated with the process.

4

Thanks for the response @warkolm! I am wondering if this is the case. I have a few gb in cache. Would that memory get attributed to the logstash process in top?

5

Thanks for the response @Badger . I ran the pmap command, but it kind of muddied the waters for me a little bit as I have never used it before. Most of the entries are mapped to [ anon ]. Though that could be indicative as something as I have a 13 page print out when I execute the query and probably 80% are attributed to anon.

I also found this formula, but am still struggling to find information on all of the different variables.
JVM memory = Heap memory+ Metaspace + CodeCache + (ThreadStackSize * Number of Threads) + DirectByteBuffers + Jvm-native

Thanks again.

6

[anon] means it is not a mapped file, not Sys V shared memory, not heap, and not stack.

How much memory does your server have?

7

Thanks for getting back to me, and explaining [ anon ].

I have 6 logstash servers and I am seeing this behavior on all of them. They all have 15gb and I have set the jvm.options to 8gb for each of the servers.

I have also seen some stuff suggesting jmap or jcmd for getting a little better information on what is going on within the jvm. I have run both, but am still researching in order to make any sense of what the outputs are telling me.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.