Java error when too many conditions in Logstash

Elastic Stack Logstash
1

I have a source (CSV) that has one field called EVENTNO. In that field are numbers (INT).
In another document I have what EVENTNO number means (e.g. "1" = "The machine is off", "2" = "The machine is on", and so on). That document has more than 22,000 entries. One entry by every possible EVENTNO value.
I want to have the meaning of the EVENTNO field instead of just the number, so what I tried was to use patterns. Something like:

filter
{
	csv { separator => ";" columns => ["DEVICEID","TIMESTAMP","MESSAGENO","ORGMESSAGE","SERVERTIMESTAMP","DEVICESTATE","EVENTNO","EVENTCOUNT","EVENTGROUPID"]}
		grok 
		{
                	match => 
			[
EVENTNO, "%{eventno_0}",
EVENTNO, "%{eventno_1}",
EVENTNO, "%{eventno_2}",
...

and then, in the patterns folder I have a file like this:

eventno_0 Yes
eventno_1 The machine is off 
eventno_2 The machine is on
eventno_3 hello 
...

I ran it and after 45 minutes just waiting and consuming a lot of memory (I change the memory limits to 8G in a 16G machine) I got a java heap error.
Then I try another aproach:

filter
{
    csv { separator => ";" columns => ["DEVICEID","TIMESTAMP","MESSAGENO","ORGMESSAGE","SERVERTIMESTAMP","DEVICESTATE","EVENTNO","EVENTCOUNT","EVENTGROUPID"]}
    if [EVENTNO] =="0" { mutate { update => {"EVENTNO" =>"Yes" } } }
    else if [EVENTNO] =="1" { mutate { update => {"EVENTNO" =>"The machine is off" } } }
    else if [EVENTNO] =="2" { mutate { update => {"EVENTNO" =>"The machine is on" } } }

Uses a lot less memory, but at the ent I get errors like:

[2020-02-28T11:30:49,709][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::StackOverflowError", :message=>"", :backtrace=>["java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1038)", "java.lang.ClassLoader.loadClass(ClassLoader.java:406)", "java.lang.ClassLoader.loadClass(ClassLoader.java:357)", "usr.share.logstash.logstash_minus_core.lib.logstash.compiler.lscl.RUBY$method$javaify_sexpr$0(/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:273)",
...

At the end, in order to be sure that the problem is memory and not another thing, I just put 10 entries (10 "mutate") and worked perfectly.
So, my question is: What is a better way to translate those 22,000 possible EVENTNO?

2

Use a translate filter.

1 Like
3

LOL... When I wrote the word "translate" I thought.. "there is my answer"... but I was at the end of my post.. Thanks @Badger!

4

That was it. I used translate filter and that did the charm.
Fast and not much memory used.
Thanks!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.