Ruby filter inline code keeps logstash to restart

Elastic Stack Logstash
1

I have a local docker ELK stack OSS image running and it works.
Version: 6.5.4
When deployed into the enterprise ELK stack which is 6.5.4, it crashes the logstash instance.

Error:

[2019-10-21T10:58:20,568][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `to_i' for [284, 0]:Array
Did you mean?  to_h
               to_a
               to_s

[2019-10-21T10:58:20,699][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"undefined method `>' for nil:NilClass", "backtrace"=>["(eval):8760880:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):8760878:in `block in initialize'", "(eval):8760903:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):8760895:in `block in initialize'", "(eval):237947:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:320:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"], :thread=>"#<Thread:0x6eb15724@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:157 sleep>"}

[2019-10-21T10:58:20,961][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `>' for nil:NilClass>, :backtrace=>["(eval):8760880:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):8760878:in `block in initialize'", "(eval):8760903:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):8760895:in `block in initialize'", "(eval):237947:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:320:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"]}

[2019-10-21T10:58:21,095][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

[2019-10-21T10:58:37,143][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.4"}

Ruby code in logstash filter is as below.

			ruby {
			    code => 'event.set("alertmeval",event.get("CurrentVal").to_i - (event.get("MaxValAllowed").to_i*0.7))'
					}
mutate {
				convert => {
					"CurrentVal" => "integer"
					"MaxValAllowed" => "integer"
				}
			}

MaxValAllowed can be from 5000 to 200000.

2

OK, so either [CurrentVal] or [MaxValAllowed] is an array containing the values [284, 0]. You would have to select an item from the array to call .to_i on it.

If you are complaining that such a minor error causes logstash to have to restart, which is very expensive, I would agree with you. I believe there are open issues around exception handling in the java execution engine. The ruby engine was much better at catching these and just restarting the pipeline.

3

Interestingly, the data is not an array. Will need to identify the record which is causing it.

By adding [0] should be ok as below?

ruby {
   code => 'event.set("alertmeval",event.get("CurrentVal")[0].to_i - (event.get("MaxValAllowed")[0].to_i*0.7))'
					}
4

That will break when it is not an array. You could try something like

CurrentVal = event.get("CurrentVal")
if CurrentVal.is_a? Array
    CurrentVal = CurrentVal[0]
end

I haven't tested that though.

5

@Badger I looked at the data, and did not have any Array result. Inserted the data without the ruby code, to view in elastic.
Used SQL Query to get field values but would not find.

Is there any other way to debug? Or in logstash another option to write a math formula.

6

There is a math filter. You might need to install it.

7

@Badger i was able to correct the ruby code. Thanks for the pointer nil.
If a field is not defined, and used in condition ruby filter throws error.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.