Ruby dumps tons of runtime errors after upgrade to 8.16.2

Elastic Stack Logstash
1

Hi everyone,
After upgrading Logstash to 8.16.2 , The /var/log/logstash/logstash-plain.log is filling up very quickly with ERROR logs pointing to what seems to be runtime errors in Ruby code. We use Ruby in one of the filtering plugin

The error messages are big in size, hence /var partition runs out of space almost daily.

Research didn't return any clue about this error.

I tried removed the Ruby code for testing, and that stopped the error message, however the code is needed.

  1. how to disable logging of that log message. I tried tuning logstash.yml and log4j2.properties but it seems that Logstash doesn't follow the logging settings from Log4j

  2. how to troubleshoot and trace the Ruby execution ?

Below is one sample error message and snippet of the filter code:
[2025-02-19T14:43:53,170][ERROR][logstash.filters.ruby ] ce)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset7.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset8.compute(Unknown Source)", "org.logstash.generated.CompiledDataset7.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset12.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset8.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset24.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset24.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset24.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset25.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset12.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset15.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset15.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset7.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset7.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset8.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:364)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:358)", "org.logstash.execution.ObservedExecution.lambda$compute$0(ObservedExecution.java:17)", "org.logstash.execution.WorkerObserver.lambda$observeExecutionComputation$0(WorkerObserver.java:39)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(ConcurrentLiveTimerMetric.java:47)", "org.logstash.execution.WorkerObserver.lambda$executeWithTimers$1(WorkerObserver.java:50)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(ConcurrentLiveTimerMetric.java:47)", "org.logstash.execution.WorkerObserver.executeWithTimers(WorkerObserver.java:50)", "org.logstash.execution.WorkerObserver.observeExecutionComputation(WorkerObserver.java:38)", "org.logstash.execution.ObservedExecution.compute(ObservedExecution.java:17)", "org.logstash.execution.WorkerLoop.abortableCompute(WorkerLoop.java:113)", "org.logstash.execution.WorkerLoop.run(WorkerLoop.java:86)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:313)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}

Original code in LS filter plugin (Ruby):

 ruby {
    id => "xx"
    code => "
      if event.get('[source][ip]').is_a?(Array)
        event.set('[source][ip]', event.get('[source][ip]').first)
      end
      if event.get('[destination][ip]').is_a?(Array)
        event.set('[destination][ip]', event.get('[destination][ip]').first)
      end
    "
  }

##################

  if [logsource_ip] {
    ruby {
    id   => "yy"
    code => "
     logsource_ip = event.get('logsource_ip')
     if logsource_ip.kind_of?(Array) then
        event.set('logsource_ip',logsource_ip[0])
        #(event['tags'] ||= []) << 'dedup_logsource_ip'
        event.tag('dedup_logsource_ip')
     end
    "
    }
}

I tried changing the code to be as following, but the error persisted


ruby {
    id => "xx"
    code => "
      source_ip = event.get('[source][ip]')
      if source_ip && source_ip.kind_of?(Array) && !source_ip.empty?
      #if event.get('[source][ip]').is_a?(Array)
        event.set('[source][ip]', event.get('[source][ip]').first)
      end
      destination_ip = event.get('[destination][ip]')
      if destination_ip && destination_ip.kind_of?(Array) && !destination_ip.empty?
      #if event.get('[destination][ip]').is_a?(Array)
        event.set('[destination][ip]', event.get('[destination][ip]').first)
      end
    "
  }
  if [logsource_ip] {
    ruby {
   id => "yy"
    code => "
     logsource_ip = event.get('logsource_ip')
     if logsource_ip && logsource_ip.kind_of?(Array) && !logsource_ip.empty?
     #if logsource_ip.kind_of?(Array) then
        event.set('logsource_ip',logsource_ip[0])
        #(event['tags'] ||= []) << 'dedup_logsource_ip'
        event.tag('dedup_logsource_ip')
     end
    "
    }
2

Hello

Upgrade from which version? What was the version you were using before.

Also, can you share your full pipeline configuration and some sample documents so this can be replicated?

3

@leandrojmp
upgrade from 8.13.3
Can't post full config.
It is basically:
000-input.conf -> 010-filter-paloalto.conf -> 900-IP-Enrichment.conf -> 999-output.conf

The ruby code is only in 900

For now i want to disable logging of the log message i shared, until i fix the issue from the root. How to disable logging for that ? I tried to tweak log4j conf file assuming that logstash uses log4j to dump runtime logs, but it seems logstash doesn't follow log4j logging settings.

4

So this is not a breaking change from 7.X, I would not expect any issue between minor versions, but would need to check every changelog between old and current version.

Unfortunately without the rest of the config and some example it is not possible to test to try to replicate.

But if you want to disable the log you could tell logstash to only log events for the ruby filter when the log level is FATAL for example:

Just run this in your logstash server.

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'{ "logger.logstash.filters.ruby" : "FATAL"}'
5

Thanks @leandrojmp , that curl command stopped the unwanted logs.
However, which file does this config line map to ? and is it persistent across reloads.
I checked in /etc/logstash/logstash.yml and /etc/logstash/log4j2.properties but the line is not there

6

This does not map to any file, this change the log level for the plugin on directly on the running service.

If you restart ou stop/start logstash you will need to run this request again.

I make no changes on the default log4j2.properties file regarding logs, I find way easier to change the log levels using the API.

7

You can make an equivalent change in log4j2.properties. Last time I did this it was something like

logger.logstash1.name = logstash.filters.ruby
logger.logstash1.level = FATAL
#logger.logstash2.name = logstash.pipeline
#logger.logstash2.level = WARN

The names logstash1/logstash2 etc do not matter.

1 Like