We are seeing Logstash dying with an out of memory error after running for anywhere from 5 to 15 minutes. We are using Logstash 6.2.2 with the following configuration:
input {
beats {
include_codec_tag => false
port => 5044
}
}
input {
file {
add_field => {
"role" => "logstash_app"
"profile" => "shipper_beats"
}
path => "/var/log/logstash/logstash-json.log"
type => "logstash"
}
}
output {
sqs {
queue => "logstash"
region => "us-east-1"
}
}
Our logstash.yml file looks like this:
---
path.data: "/var/lib/logstash"
path.config: "/etc/logstash/conf.d"
path.logs: "/var/log/logstash"
dead_letter_queue.enable: true
log.format: json
node.name: i-005f2e0f19b7e9839
pipeline.workers: 12
Our jvm.options:
-Dfile.encoding=UTF-8
-Djava.awt.headless=true
-XX:+DisableExplicitGC
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=75
-Xms256m
-Xmx1g
From journalctl:
Feb 22 23:58:52 ip-10-181-69-15 systemd[1]: Started logstash.
Feb 22 23:59:13 ip-10-181-69-15 logstash[18309]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Feb 22 23:59:24 ip-10-181-69-15 logstash[18309]: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/documenter/has_many_operation_documenter.rb:7: warning: `+' after local variable or literal is interpreted as binary operator
Feb 22 23:59:24 ip-10-181-69-15 logstash[18309]: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/documenter/has_many_operation_documenter.rb:7: warning: even though it seems like unary operator
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: java.lang.OutOfMemoryError: Java heap space
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: Dumping heap to java_pid18309.hprof ...
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: Unable to create java_pid18309.hprof: Permission denied
I can provide a heap dump, but it contains log entries that are somewhat sensitive and so I am not sure how best to share it.