Recently upgraded to Elastic 8.16.0, Kibana 8.16.0 and logstash 8.16.0. And things were running smoothly until Friday morning at 1am. Here in the States it was Thanksgiving so I did not change anything recently. System is a RedHat Enterprise VM with ~128GB RAM and plenty of disk space. Upgrades to Elastic and Kibana were done via yum and logstash was updated via tarball download and unpack (this wasn't my installation, I am the one who has inherited it).
I am receiving the following error:
[2024-12-01T22:09:26,951][WARN ][logstash.inputs.jdbc ][metrics][ec982323546e6f3c9f890e6df50001f6978d13d3bb47dfb7e4bac6c98c5a1edc] Exception when executing JDBC query {:exception=>Sequel::DatabaseError, :message=>"Java::JavaSql::SQLException: Java heap space", :cause=>"#<Java::JavaSql::SQLException: Java heap space
I've increased heap space to 4GB (from 1GB).
This error occurs over and over until logstash restarts itself. Almost all of our data is obtained via the jdbc plugin, so metrics haven't been updated since Friday.
Is there a simple cause for this that I'm missing (which wouldn't surprise me, given my still-developing knowledge of ELK administration)? I have pored over many other threads but haven't been able to come up with a solution.
I also see these errors:
Dec 01 22:15:52 2024 [2024-12-01T22:15:41,911][FATAL][org.logstash.Logstash ][metrics][40a23460c92b7bde8eebba48f5e3dcd69a40075e512d0ac0a81832e2df072e3b] uncaught error (in thread Ruby-0-Thread-190@[metrics]|input|Jdbc|scheduler: /proj/emu/metrics/logstash/logstash-8.16.0/vendor/bundle/jruby/3.1.0/gems/rufus-scheduler-3.9.2/lib/rufus/sc│
De│Dec 01 22:15:52 2024 java.lang.OutOfMemoryError: Java heap space │
De│Dec 01 22:15:52 2024 [2024-12-01T22:15:41,909][FATAL][org.logstash.Logstash ][metrics] uncaught error (in thread Ruby-0-Thread-80: /proj/emu/metrics/logstash/logstash-8.16.0/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.9-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:235) │
De│Dec 01 22:15:52 2024 java.lang.OutOfMemoryError: Java heap space │
De│Dec 01 22:15:52 2024 [2024-12-01T22:15:41,915][FATAL][org.logstash.Logstash ][metrics] uncaught error (in thread Ruby-0-Thread-13: /proj/emu/metrics/logstash/logstash-8.16.0/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.9-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:235) │
De│Dec 01 22:15:52 2024 java.lang.OutOfMemoryError: Java heap space │
De│Dec 01 22:15:52 2024 at java.lang.Long.valueOf(Long.java:1206) ~[?:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyThread$SleepTask2.run(RubyThread.java:1717) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyThread$SleepTask2.run(RubyThread.java:1701) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyThread.executeTask(RubyThread.java:1770) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyThread.executeTaskBlocking(RubyThread.java:1744) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyThread.sleep(RubyThread.java:1618) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyKernel.sleepCommon(RubyKernel.java:860) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyKernel.sleep(RubyKernel.java:845) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyKernel$INVOKER$s$sleep.call(RubyKernel$INVOKER$s$sleep.gen) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:242) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:61) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:297) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:103) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:545) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.runtime.Block.call(Block.java:144) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.RubyProc.call(RubyProc.java:354) ~[jruby.jar:?] │
De│Dec 01 22:15:52 2024 at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111) ~[jruby.jar:?] │am
-r│Dec 01 22:15:52 2024 at java.lang.Thread.runWith(Thread.java:1596) ~[?:?] │oo
ls│Dec 01 22:15:52 2024 at java.lang.Thread.run(Thread.java:1583) ~[?:?]
But I'm assuming these are all related to the heap space issue.
Sometimes, if the error isn't the above, it's the following:
[2024-12-01T21:52:04,745][ERROR][logstash.inputs.jdbc ][metrics][3fd7851ddb7de636663504c08670cb515c82a051448b8b6d53569dd83cbacba0] Java::JavaSql::SQLException: Can not read response from server. Expected to read 1,904 bytes, read 361 bytes before connection was unexpectedly lost.: SELECT
Any articles you could forward me to or help you could provide would be much appreciated.
Thank you!