If/Else comparison to add tag

vvwood · September 15, 2020, 3:41pm

Hey y'all,
I have been beating my head against the wall trying to accomplish something that seemed rather simple. If anyone could spot would I am missing, I would appreciate it. The seemingly simple task is to compare value of a field to a number using an if, else if, else setup and then add a tag to the document depending on where true is evaluated. Expanding this... I have extracted the hour from @timestamp to a field called "hour". I now want to evaluate each document as they are being processed by the logstash pipeline and add a tag depending on whether that document occurred during or after business hours. Once the tag is added, I will be using that as a filter within dashboards.

I have 6 different attempts at this which all resulted in an error causing the pipeline to be perpetually restarted or just stopped. After reviewing the following, any helpful thoughts?

if ([Hour] < 11) {
            mutate {
                add_tag => "AfterHoursBadgeSwipe"
            }
        } else if ([Hour] > 21) {
            mutate {
                add_tag => "AfterHoursBadgeSwipe"
            }
        } else {
            mutate {            
                add_tag => "BusinessHoursBadgeSwipe"
            }
        }

if (doc[hour].value < '11') {
            add_tag => "AfterHoursBadgeSwipe"
        } else if (doc[hour].value > '21') {
            add_tag => "AfterHoursBadgeSwipe"
        } else {
            add_tag => "BusinessHoursBadgeSwipe"
        }

if [hour] < 11 or [hour] > 21 {
    add_tag => "AfterHoursBadgeSwipe"
} else {
    add_tag => "BusinessHoursBadgeSwipe"
}

if [hour] < "11" {
    mutate { add_tag => "AfterHoursBadgeSwipe" }
  } else if [hour] > "21" {
    mutate { add_tag => "AfterHoursBadgeSwipe" }
  } else {
    add_tag => "BusinessHoursBadgeSwipe"
  }

MISSING SQUARE BRACKETS MAYBE?
if [hour] < "11" {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else if [hour] > "21" {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else {
        mutate { add_tag => ["BusinessHoursBadgeSwipe"] }
    }

NEXT...
    mutate {
        convert => { "hour" => "integer" }
    }
    if [hour] < 11 {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else if [hour] > 21 {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else {
        mutate { add_tag => ["BusinessHoursBadgeSwipe"] }
    }

ylasri · September 15, 2020, 3:47pm

May be try first to convert your field into numeric and that was your last solution
Can you share the error log ?

filter {
  mutate {
    convert => { "hour" => "integer" }
  }
}

vvwood · September 15, 2020, 6:45pm

Here are the same configs but including the errors inline.

THIS DOESN'T WORK:
if ([Hour] < 11) {
            mutate {
                add_tag => "AfterHoursBadgeSwipe"
            }
        } else if ([Hour] > 21) {
            mutate {
                add_tag => "AfterHoursBadgeSwipe"
            }
        } else {
            mutate {            
                add_tag => "BusinessHoursBadgeSwipe"
            }
        }
THIS EITHER
if (doc[hour].value < '11') {
            add_tag => "AfterHoursBadgeSwipe"
        } else if (doc[hour].value > '21') {
            add_tag => "AfterHoursBadgeSwipe"
        } else {
            add_tag => "BusinessHoursBadgeSwipe"
        }
MAYBE THIS
if [hour] < 11 or [hour] > 21 {
    add_tag => "AfterHoursBadgeSwipe"
} else {
    add_tag => "BusinessHoursBadgeSwipe"
}
RESULT
[2020-09-15T10:42:05,512][ERROR][logstash.agent           ] Failed to execute action {:id=>:LOGSOURCENAME, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at line 20, column 17 (byte 653) after filter {\n    csv {\n\t\tsource => \"message\"\n\t\tcolumns => [ \"Sequence\",\"Date and Time\",\"Event message\",\"Event number\",\"Object #1\",\"Description #1\",\"Object #2\",\"Description #2\",\"Object #3\",\"Description #3\",\"Object #4\",\"Description #4\",\"Card number\"]\n        separator => \",\"\n    }\n    date {\n        match => [ \"Date and Time\", \"MM/dd/yyyy hh:mm:ss a\", \"M/d/yyyy hh:mm:ss a\", \"M/dd/yyyy hh:mm:ss a\", \"M/dd/yyyy hh:mm a\" ]\n        timezone => \"America/New_York\"\n        target => \"@timestamp\"\n        add_field => {\"[hour]\" => \"%{+kk}\"}\n    }\n    if [hour] < 11 or [hour] > 21 {\n        add_tag ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:356:in `block in converge_state'"]}
PER ELASTIC DOCS
if [hour] < "11" {
    mutate { add_tag => "AfterHoursBadgeSwipe" }
  } else if [hour] > "21" {
    mutate { add_tag => "AfterHoursBadgeSwipe" }
  } else {
    add_tag => "BusinessHoursBadgeSwipe"
  }
RESULT
[2020-09-15T10:49:32,837][ERROR][logstash.agent           ] Failed to execute action {:id=>:LOGSOURCENAME, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any filter plugin named 'else'. Are you sure this is correct? Trying to load the else filter plugin resulted in this error: no such file to load -- logstash/filters/else", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:80)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.reload.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:356)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]}
MISSING SQUARE BRACKETS MAYBE?
if [hour] < "11" {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else if [hour] > "21" {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else {
        mutate { add_tag => ["BusinessHoursBadgeSwipe"] }
    }
[2020-09-15T11:05:51,149][ERROR][logstash.javapipeline    ][LOGSOURCENAME] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"LOGSOURCENAME", :error=>"", :exception=>Java::JavaLang::NullPointerException, :backtrace=>["org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679)", "org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453)", "org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444)", "org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47)", "org.logstash.generated.CompiledDataset53.compute(Unknown Source)", "org.logstash.generated.CompiledDataset54.compute(Unknown Source)", "org.logstash.generated.CompiledDataset58.compute(Unknown Source)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:339)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:333)", "org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83)", "sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)", "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:293)", "org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24)", "org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:289)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x44da48f6 sleep>"}
NEXT...
    mutate {
        convert => { "hour" => "integer" }
    }
    if [hour] < 11 {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else if [hour] > 21 {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else {
        mutate { add_tag => ["BusinessHoursBadgeSwipe"] }
    }
RESULT 
[2020-09-15T11:19:12,515][ERROR][logstash.javapipeline    ][LOGSOURCENAME] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"LOGSOURCENAME", :error=>"", :exception=>Java::JavaLang::NullPointerException, :backtrace=>["org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679)", "org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453)", "org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444)", "org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47)", "org.logstash.generated.CompiledDataset59.compute(Unknown Source)", "org.logstash.generated.CompiledDataset60.compute(Unknown Source)", "org.logstash.generated.CompiledDataset61.compute(Unknown Source)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:339)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:333)", "org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83)", "sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)", "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)", "org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:289)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x475dfc94 sleep>"}

Badger · September 15, 2020, 6:57pm

Are you certain the field always exists? See this thread.

vvwood · September 15, 2020, 7:56pm

I do not believe this is a case of the field not existing. The field that I am comparing is created during logstash processing from the time in the timestamp. I'll show an expanded view of the config here showing the creation of "hour" and you can think about the result of it.

date {
        match => [ "Date and Time", "MM/dd/yyyy hh:mm:ss a", "M/d/yyyy hh:mm:ss a", "M/dd/yyyy hh:mm:ss a", "M/dd/yyyy hh:mm a" ]
        timezone => "America/New_York"
        target => "@timestamp"
        add_field => {"[hour]" => "%{+HH}"}
    }
    mutate {
        convert => { "hour" => "integer" }
    }
    if [hour] < 11 {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else if [hour] > 21 {
        mutate { add_tag => ["AfterHoursBadgeSwipe"] }
    } else {
        mutate { add_tag => ["BusinessHoursBadgeSwipe"] }
    }

I did a cursory review just to be sure and all of the last few days of sample docs have the hour field populated and the convert to integer succeeded. I am now checking to see if moving the if/else further down in the config makes a difference.

-VW

vvwood · September 15, 2020, 8:01pm

Confirmed! (Deleted index & re ran ingest with if/else statement resulting in NullPointerException)
The problem is definitely to do with the if/else syntax but I am not sure what.
-VW

system · October 13, 2020, 8:01pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.