Logstash 2.3.4 memory leak

Elastic Stack Logstash
1

Hi!

I've been working on setting up ELK in AWS and been having this issue with Constantly increasing consumption of memory. I started out with 4GB then up to 10GB! But yes, it still consumed causing the logstash server to crash. restarting the server fixes the problem temporarily. But it just won't go away.

I have a basic input configuration using the cloudwatch logs input plugin. with 9 Log groups as inputs and uses logstash output plugins for elastic search. We use the elasticsearch server version 1.5 from AWS.

Hope you guys can give a hint on how to fix the problem thanks!

2

By the way the Logstash server runs on top of "Ubuntu 16.04.1 LTS".

Thanks!

3

Providing your config would be helpful.

4

Hi Mark Walkom,

Thanks for responding to my queries. Here is the basic config that I have:

INPUTS: this is the same for 9 cloudwatch log groups

========================================================

input {
cloudwatch_logs {
log_group => "tomcatxxx_server_log"
sincedb_path => "/dev/null"
interval=> "300"
region => "us-east-1"
}
}

=======================================================================

OUTPUT:

output{
elasticsearch {
hosts => "search-es-domain.us-east-1.es.amazonaws.com:80"
ssl => false
flush_size => 700
}
}

Thanks!

5

Can you expand on what you mean by memory leak? Is this exceeding the heap you are setting?

6

Hi Mark,

Yes, it is exceeding the heap settings that I configured on the logstash
server. I started with 2GB then 8GB and right now it's set to 10GB and it
is consumed in just a span of 30 mins.

Thanks!

7

Is this JVM heap though, or other memory?
Where are you seeing this?

8

On the OS memory usage. And when checking the service status of logstash in
ubuntu. It shows the actual memory usage of the service.

9

That's not JVM heap though, that's just the OS using all the available memory it can to make it performant as possible.
Limiting that is outside the scope of these forums.

10

But restarting the logstash server, sets the memory back to normal value. I'm sure it has something to do with logstash server it self. Also logstash server is the only service running on the particular server.

11

Logstash will only use what is in JVM heap.

12

So Is there any way to resolve the problem? The logstash server keeps on crushing right now.

Thanks!

13

Anyone with the idea please?

Thanks!

14

Anyone have an idea to resolved the memory leak??

15

You need to do something like this https://www.google.com.au/?gws_rd=ssl#q=linux+limit+process+memory

16

I'll give it a try. Thanks!