Java error when too many conditions in Logstash

I have a source (CSV) that has one field called EVENTNO. In that field are numbers (INT).
In another document I have what EVENTNO number means (e.g. "1" = "The machine is off", "2" = "The machine is on", and so on). That document has more than 22,000 entries. One entry by every possible EVENTNO value.
I want to have the meaning of the EVENTNO field instead of just the number, so what I tried was to use patterns. Something like:

                	match => 
EVENTNO, "%{eventno_0}",
EVENTNO, "%{eventno_1}",
EVENTNO, "%{eventno_2}",

and then, in the patterns folder I have a file like this:

eventno_0 Yes
eventno_1 The machine is off 
eventno_2 The machine is on
eventno_3 hello 

I ran it and after 45 minutes just waiting and consuming a lot of memory (I change the memory limits to 8G in a 16G machine) I got a java heap error.
Then I try another aproach:

    if [EVENTNO] =="0" { mutate { update => {"EVENTNO" =>"Yes" } } }
    else if [EVENTNO] =="1" { mutate { update => {"EVENTNO" =>"The machine is off" } } }
    else if [EVENTNO] =="2" { mutate { update => {"EVENTNO" =>"The machine is on" } } }

Uses a lot less memory, but at the ent I get errors like:

[2020-02-28T11:30:49,709][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::StackOverflowError", :message=>"", :backtrace=>["java.lang.ClassLoader.findLoadedClass(", "java.lang.ClassLoader.loadClass(", "java.lang.ClassLoader.loadClass(", "usr.share.logstash.logstash_minus_core.lib.logstash.compiler.lscl.RUBY$method$javaify_sexpr$0(/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:273)",

At the end, in order to be sure that the problem is memory and not another thing, I just put 10 entries (10 "mutate") and worked perfectly.
So, my question is: What is a better way to translate those 22,000 possible EVENTNO?

Use a translate filter.

1 Like

LOL... When I wrote the word "translate" I thought.. "there is my answer"... but I was at the end of my post.. Thanks @Badger!

That was it. I used translate filter and that did the charm.
Fast and not much memory used.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.