# "java.lang.OutOfMemoryError: Java heap space" after running for a few minutes

**URL:** <https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-after-running-for-a-few-minutes/121165>\
**Category:** Logstash\
**Created:** [February 23, 2018, 5:04am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-after-running-for-a-few-minutes/121165 "2018-02-23T05:04:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![joshuaspence](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshuaspence/32/10939_2.png) [@joshuaspence](https://discuss.elastic.co/u/joshuaspence)\
**Post date:** [February 23, 2018, 5:04am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-after-running-for-a-few-minutes/121165/1 "2018-02-23T05:04:25Z")

</div>

We are seeing Logstash dying with an out of memory error after running for anywhere from 5 to 15 minutes. We are using Logstash 6.2.2 with the following configuration:

```auto
input {
  beats {
    include_codec_tag => false
    port => 5044
  }
}
input {
  file {
    add_field => {
      "role" => "logstash_app"
      "profile" => "shipper_beats"
    }
    path => "/var/log/logstash/logstash-json.log"
    type => "logstash"
  }
}
output {
  sqs {
    queue => "logstash"
    region => "us-east-1"
  }
}

```

Our `logstash.yml` file looks like this:

```auto
---
path.data: "/var/lib/logstash"
path.config: "/etc/logstash/conf.d"
path.logs: "/var/log/logstash"
dead_letter_queue.enable: true
log.format: json
node.name: i-005f2e0f19b7e9839
pipeline.workers: 12

```

Our `jvm.options`:

```auto
-Dfile.encoding=UTF-8
-Djava.awt.headless=true
-XX:+DisableExplicitGC
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=75
-Xms256m
-Xmx1g

```

From `journalctl`:

```auto
Feb 22 23:58:52 ip-10-181-69-15 systemd[1]: Started logstash.
Feb 22 23:59:13 ip-10-181-69-15 logstash[18309]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Feb 22 23:59:24 ip-10-181-69-15 logstash[18309]: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/documenter/has_many_operation_documenter.rb:7: warning: `+' after local variable or literal is interpreted as binary operator
Feb 22 23:59:24 ip-10-181-69-15 logstash[18309]: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/documenter/has_many_operation_documenter.rb:7: warning: even though it seems like unary operator
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: java.lang.OutOfMemoryError: Java heap space
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: Dumping heap to java_pid18309.hprof ...
Feb 23 00:01:53 ip-10-181-69-15 logstash[18309]: Unable to create java_pid18309.hprof: Permission denied

```

I can provide a heap dump, but it contains log entries that are somewhat sensitive and so I am not sure how best to share it.

---

<div class="post-metadata">

**Author:** ![joshuaspence](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshuaspence/32/10939_2.png) [@joshuaspence](https://discuss.elastic.co/u/joshuaspence)\
**Post date:** [February 23, 2018, 5:10am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-after-running-for-a-few-minutes/121165/2 "2018-02-23T05:10:05Z")

</div>

I thought that it might be related to [https://github.com/logstash-plugins/logstash-input-beats/issues/286](https://github.com/logstash-plugins/logstash-input-beats/issues/286), but I updated `logstash-input-beats` to the latest version and it didn't seem to help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2018, 5:10am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-after-running-for-a-few-minutes/121165/3 "2018-03-23T05:10:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.