"java.lang.OutOfMemoryError: Java heap space" after running for a few minutes


(Joshuaspence) #1

We are seeing Logstash dying with an out of memory error after running for anywhere from 5 to 15 minutes. We are using Logstash 6.2.2 with the following configuration:

input {
  beats {
    include_codec_tag => false
    port => 5044
  }
}
input {
  file {
    add_field => {
      "role" => "logstash_app"
      "profile" => "shipper_beats"
    }
    path => "/var/log/logstash/logstash-json.log"
    type => "logstash"
  }
}
output {
  sqs {
    queue => "logstash"
    region => "us-east-1"
  }
}

Our logstash.yml file looks like this:

---
path.data: "/var/lib/logstash"
path.config: "/etc/logstash/conf.d"
path.logs: "/var/log/logstash"
dead_letter_queue.enable: true
log.format: json
node.name: i-005f2e0f19b7e9839
pipeline.workers: 12

Our jvm.options:

-Dfile.encoding=UTF-8
-Djava.awt.headless=true
-XX:+DisableExplicitGC
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=75
-Xms256m
-Xmx1g

From journalctl:

Feb 22 23:58:52 ip-10-181-69-15 systemd[1]: Started logstash.
Feb 22 23:59:13 ip-10-181-69-15 logstash[18309]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Feb 22 23:59:24 ip-10-181-69-15 logstash[18309]: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/documenter/has_many_operation_documenter.rb:7: warning: `+' after local variable or literal is interpreted as binary operator
Feb 22 23:59:24 ip-10-181-69-15 logstash[18309]: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/documenter/has_many_operation_documenter.rb:7: warning: even though it seems like unary operator
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: java.lang.OutOfMemoryError: Java heap space
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: Dumping heap to java_pid18309.hprof ...
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: Unable to create java_pid18309.hprof: Permission denied

I can provide a heap dump, but it contains log entries that are somewhat sensitive and so I am not sure how best to share it.


(Joshuaspence) #2

I thought that it might be related to https://github.com/logstash-plugins/logstash-input-beats/issues/286, but I updated logstash-input-beats to the latest version and it didn't seem to help.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.