# Error: Your application used more memory than the safety cap of 4G **URL:** \ **Category:** Logstash\ **Created:** [January 12, 2018, 11:46am UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285 "2018-01-12T11:46:02Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jose\_rebelo](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@jose\_rebelo](https://discuss.elastic.co/u/jose_rebelo)\ **Post date:** [January 12, 2018, 11:46am UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/1 "2018-01-12T11:46:03Z")
Hello, we have are dealing with this error since ever... (basically after installing and start feeding it) We changed jmx and jms from initial 1Gb to 2Gb and then to 4Gb and we are consistently dealing with this error almost ever at the same clock time... If we restart ELK at 9-10 AM then we will have logstash crashing with this error around 11PM. > Error: Your application used more memory than the safety cap of 4G. > Specify -J-Xmx####M to increase it (#### = cap size in MB). > Specify -w for full java.lang.OutOfMemoryError: Java heap space stack trace We are using Jenkins logstash plugin (1.2.0) to send data to logstash (6.0.0). The logstash plugins versions we have are the following: > logstash-codec-cef (5.0.1-java) > logstash-codec-collectd (3.0.7) > logstash-codec-dots (3.0.5) > logstash-codec-edn (3.0.5) > logstash-codec-edn\_lines (3.0.5) > logstash-codec-es\_bulk (3.0.5) > logstash-codec-fluent (3.1.4-java) > logstash-codec-graphite (3.0.4) > logstash-codec-json (3.0.4) > logstash-codec-json\_lines (3.0.4) > logstash-codec-line (3.0.4) > logstash-codec-msgpack (3.0.6-java) > logstash-codec-multiline (3.0.7) > logstash-codec-netflow (3.7.0) > logstash-codec-plain (3.0.4) > logstash-codec-rubydebug (3.0.4) > logstash-filter-aggregate (2.7.0) > logstash-filter-anonymize (3.0.5) > logstash-filter-cidr (3.1.1-java) > logstash-filter-clone (3.0.4) > logstash-filter-csv (3.0.6) > logstash-filter-date (3.1.8) > logstash-filter-de\_dot (1.0.2) > logstash-filter-dissect (1.1.1) > logstash-filter-dns (3.0.6) > logstash-filter-drop (3.0.4) > logstash-filter-elasticsearch (3.2.0) > logstash-filter-fingerprint (3.1.1) > logstash-filter-geoip (5.0.1-java) > logstash-filter-grok (3.4.3) > logstash-filter-jdbc\_streaming (1.0.2) > logstash-filter-json (3.0.4) > logstash-filter-kv (4.0.2) > logstash-filter-metrics (4.0.4) > logstash-filter-mutate (3.1.6) > logstash-filter-ruby (3.0.4) > logstash-filter-sleep (3.0.5) > logstash-filter-split (3.1.4) > logstash-filter-syslog\_pri (3.0.4) > logstash-filter-throttle (4.0.3) > logstash-filter-translate (3.0.3) > logstash-filter-truncate (1.0.3) > logstash-filter-urldecode (3.0.5) > logstash-filter-useragent (3.2.1-java) > logstash-filter-xml (4.0.4) > logstash-input-beats (5.0.2-java) > logstash-input-dead\_letter\_queue (1.1.1) > logstash-input-elasticsearch (4.1.0) > logstash-input-exec (3.1.4) > logstash-input-file (4.0.3) > logstash-input-ganglia (3.1.2) > logstash-input-gelf (3.0.6) > logstash-input-generator (3.0.4) > logstash-input-graphite (3.0.4) > logstash-input-heartbeat (3.0.4) > logstash-input-http (3.0.6) > logstash-input-http\_poller (4.0.3) > logstash-input-imap (3.0.4) > logstash-input-jdbc (4.3.0) > logstash-input-kafka (8.0.2) > logstash-input-pipe (3.0.5) > logstash-input-rabbitmq (6.0.1) > logstash-input-redis (3.1.5) > logstash-input-s3 (3.1.7) > logstash-input-snmptrap (3.0.4) > logstash-input-sqs (3.0.5) > logstash-input-stdin (3.2.4) > logstash-input-syslog (3.2.2) > logstash-input-tcp (5.0.2-java) > logstash-input-twitter (3.0.6) > logstash-input-udp (3.1.2) > logstash-input-unix (3.0.5) > logstash-mixin-aws (4.2.3) > logstash-mixin-http\_client (6.0.1) > logstash-mixin-rabbitmq\_connection (5.0.0-java) > logstash-output-cloudwatch (3.0.6) > logstash-output-csv (3.0.5) > logstash-output-elasticsearch (9.0.0-java) > logstash-output-email (4.0.6) > logstash-output-file (4.1.1) > logstash-output-graphite (3.1.3) > logstash-output-http (5.1.0) > logstash-output-kafka (7.0.4) > logstash-output-lumberjack (3.1.5) > logstash-output-nagios (3.0.4) > logstash-output-null (3.0.4) > logstash-output-pagerduty (3.0.5) > logstash-output-pipe (3.0.4) > logstash-output-rabbitmq (5.0.2-java) > logstash-output-redis (4.0.2) > logstash-output-s3 (4.0.12) > logstash-output-sns (4.0.5) > logstash-output-sqs (5.0.1) > logstash-output-stdout (3.1.2) > logstash-output-tcp (5.0.1) > logstash-output-udp (3.0.4) > logstash-output-webhdfs (3.0.4) > logstash-patterns-core (4.1.2) We have seen many old reports about this issue but for older versions of logstash that supposedly the next newer version fixed the "bug" in some multiline plugin... But in our case we using the latest version of everything!! We are using jdk 8 u 151 > openjdk version "1.8.0\_151" > OpenJDK Runtime Environment (build 1.8.0\_151-8u151-b12-0ubuntu0.16.04.2-b12) > OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) In a docker image container which is OS version: > DISTRIB\_ID=Ubuntu > DISTRIB\_RELEASE=16.04 > DISTRIB\_CODENAME=xenial > DISTRIB\_DESCRIPTION="Ubuntu 16.04.2 LTS" > NAME="Ubuntu" > VERSION="16.04.2 LTS (Xenial Xerus)" > ID=ubuntu > ID\_LIKE=debian > PRETTY\_NAME="Ubuntu 16.04.2 LTS" > VERSION\_ID="16.04" > HOME\_URL="[http://www.ubuntu.com/](http://www.ubuntu.com/)" > SUPPORT\_URL="[http://help.ubuntu.com/](http://help.ubuntu.com/)" > BUG\_REPORT\_URL="[Bugs : Ubuntu](http://bugs.launchpad.net/ubuntu/)" > VERSION\_CODENAME=xenial > UBUNTU\_CODENAME=xenial Can you please give us some hint on how to solve this issue for good? Thank you in advance. Regards, José Rebelo ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 16, 2018, 4:35pm UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/2 "2018-01-16T16:35:49Z")
What does your pipeline configuration look like? ---
**Author:** ![jose\_rebelo](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@jose\_rebelo](https://discuss.elastic.co/u/jose_rebelo)\ **Post date:** [January 19, 2018, 11:39am UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/3 "2018-01-19T11:39:45Z")
Hi there, we have the following configurations: > input { > udp { > codec =\> plain > port =\> 1514 > type =\> doc > queue\_size =\> 72000 > receive\_buffer\_bytes =\> 31457280 > buffer\_size =\> 65536 > } > } > output { > elasticsearch { > hosts =\> ["0.0.0.0:9200"] > index =\> "logstash-jenkins" > } > stdout { codec =\> rubydebug } > } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 23, 2018, 8:44pm UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/4 "2018-01-23T20:44:54Z")
Perhaps Logstash isn't processing the events fast enough so your buffers are filling up? What kind of inbound message rate do you have? Is ES keeping up? ---
**Author:** ![jose\_rebelo](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@jose\_rebelo](https://discuss.elastic.co/u/jose_rebelo)\ **Post date:** [January 24, 2018, 9:41am UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/5 "2018-01-24T09:41:57Z")
> [@magnusbaeck](#): > > Perhaps Logstash isn't processing the events fast enough so your buffers are filling up? What kind of inbound message rate do you have? Is ES keeping up? Hello, I'm not sure I'm getting your point. We are using jenkins plugin for logstash which is using syslog indexer type. The protocol is UDP (RFC5424). The messages we are sending are sent from hundreds of jenkins jobs which have hundreds if not thousands of lines... We don't have control over the logstash calls cadence because jobs are running in continuous integration mode. Don't think this has to do with ES because we are experiencing the Java Heap Space issue still in LS... Isn't that so? Meanwhile I've also noticed additional output log message that might be useful... > [@](#): > > Dumping heap to java\_pid193.hprof ... > Heap dump file created [4346989957 bytes in 31.535 secs] > [2018-01-23T13:34:03,213][ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline\_id=\>"main", "exception"=\>"Java heap space", "backtrace"=\>["java.nio.HeapCharBuffer.(HeapCharBuffer.java:57)", "java.nio.CharBuffer.allocate(CharBuffer.java:335)", "org.jruby.RubyEncoding$UTF8Coder.(RubyEncoding.java:272)", "org.jruby.RubyEncoding.getUTF8Coder(RubyEncoding.java:333)", "org.jruby.RubyEncoding.decodeUTF8(RubyEncoding.java:248)", "org.jruby.runtime.Helpers.decodeByteList(Helpers.java:2659)", "org.jruby.RubyString.decodeString(RubyString.java:761)", "org.jruby.RubyString.toString(RubyString.java:752)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby\_sprintf(JrubyEventExtLibrary.java:166)", "java.lang.invoke.LambdaForm$DMH/149928006.invokeSpecial\_L3\_L(LambdaForm$DMH)", "java.lang.invoke.LambdaForm$BMH/1773206895.reinvoke(LambdaForm$BMH)", "java.lang.invoke.LambdaForm$MH/258952499.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/632249781.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/258952499.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/632249781.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/352359770.linkToCallSite(LambdaForm$MH)", "opt.logstash.vendor.bundle.jruby.$2\_dot\_3\_dot\_0.gems.logstash\_minus\_filter\_minus\_aggregate\_minus\_2\_dot\_7\_dot\_2.lib.logstash.filters.aggregate.RUBY$method$filter$0(/opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-aggregate-2.7.2/lib/logstash/filters/aggregate.rb:176)", "java.lang.invoke.LambdaForm$DMH/1450495309.invokeStatic\_L7\_L(LambdaForm$DMH)", "java.lang.invoke.LambdaForm$BMH/1923232046.reinvoke(LambdaForm$BMH)", "java.lang.invoke.LambdaForm$MH/1325547227.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1134612201.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1325547227.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1134612201.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1325547227.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1134612201.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1325547227.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1134612201.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1325547227.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1134612201.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1325547227.delegate(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1134612201.guard(LambdaForm$MH)", "java.lang.invoke.LambdaForm$MH/1325547227.delegate(LambdaForm$MH)"], :thread=\>"#\"} Does this have to do with the fact we have several filter files?? I'm just saying because I see the following in the previous message: > [@](#): > > "opt.logstash.vendor.bundle.jruby.$2\_dot\_3\_dot\_0.gems.logstash\_minus\_filter\_minus\_aggregate\_minus\_2\_dot\_7\_dot\_2.lib.logstash.filters.aggregate.RUBY$method$filter$0(/opt/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-aggregate-2.7.2/lib/logstash/filters/aggregate.rb:176)", By the way, we have migrated everything to the version 6.1.2 (LS+ES+KIBANA) and the problem is the same... Regards, José Rebelo ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 25, 2018, 9:53pm UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/6 "2018-01-25T21:53:34Z")
You've configured the udp input to have an internal queue of 72k messages. If that queue fills up because Logstash isn't able to process events (possibly because ES isn't able to accept messages sufficiently quickly) then Logstash is going to use some memory. However, even if the messages are 1k each the whole queue shouldn't use that much more than 72 MB. Sure, there's overhead but hardly enough to fill a 4 GB JVM heap. Perhaps there's a memory leak somewhere? ---
**Author:** ![jose\_rebelo](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@jose\_rebelo](https://discuss.elastic.co/u/jose_rebelo)\ **Post date:** [January 26, 2018, 10:02am UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/7 "2018-01-26T10:02:49Z")
Hi, I think we have figured it out. Basically we had an exception in "logstash.stdout" (during ELK startup) stating that the file "logstash-slowlog-plain.log" was not possible to be created. (lack of permissions for ELK user in the folder where that file was supposed to be created "log") By fixing the permissions issue we have no longer that exception in log and ELK is running for already 2 days without interruption (which we never had untill now) Regards, José Rebelo ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 23, 2018, 10:03am UTC](https://discuss.elastic.co/t/error-your-application-used-more-memory-than-the-safety-cap-of-4g/115285/8 "2018-02-23T10:03:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.