Pipeline worker error in logstash config

Hello all.
I am trying to parse a VPN device logs. t send data in KV format. The data received as I see in netcat's output is-

<13>Sep 29 21:52:00 172.xx.43.101 488 <134>1 2021-09-29T21:52:00+05:30 vpn.com2 PulseSecure: - - - id=firewall time="2021-09-29 21:52:00" pri=6 fw=172.xx.43.101 vpn=ive user=user1 realm="google_auth" roles="Domain_check_role" proto= src=36.xx.87.4 dst= dstname= type=vpn op= arg="" result= sent= rcvd= agent="" duration= msg="NWC23464: VPN Tunneling: Session started for user  (session: sid48b96b810c824869195b3fc0c2244cce9292c1d900000000) with IPv4 address 172.xx.223.184, hostname host-1"

my logstash configuration is-

input {
syslog { port => 1301 ecs_compatibility => disabled tags => ["vpn"] }
}
filter {
if ["vpn"] in [tags] {
dissect { mapping => { "message" => "%{reserved}- - - %{message1}" } }
kv {
source => "message1"
value_split => "="
} }
}
output {
if "vpn" in [tags] {
elasticsearch {
hosts => "localhost"
index => "vpn-%{+YYYYMMdd}"
user => "elastic"
password => "passwordxxxx"
} }
#stdout { }
}

I am getting below error-

[ERROR] 2021-09-29 22:06:13.903 [[main]>worker2] javapipeline - Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"", :exception=>Java::JavaLang::NullPointerException, :backtrace=>["org.logstash.config.ir.compiler.EventCondition$Compiler.contains(EventCondition.java:464)", "org.logstash.config.ir.compiler.EventCondition$Compiler.access$1000(EventCondition.java:80)", "org.logstash.config.ir.compiler.EventCondition$Compiler$FieldInField.fulfilled(EventCondition.java:616)", "org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47)", "org.logstash.generated.CompiledDataset1.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:329)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:323)", "org.logstash.execution.WorkerLoop.run(WorkerLoop.java:87)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base/java.lang.reflect.Method.invoke(Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)", "org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:295)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"], :thread=>"#<Thread:0x1f982b12 sleep>"}

Could not post the complete error due to allowed characters limit. But the same error block above repeats 5 more times. and then logstash shuts down.
I've never seen this error before with KV plugin. KV plugin with same kind of configuration is running on other machine without problems.

Just for info, I ran updates on the ubuntu machine. Kibana and logstash both got updated. Kibana wasn't connecting to elasticsearch, so had to reinstall the older version.

Elasticsearch version- 7.13.3
Logstash version- 7.15.0

Well that is a spectacularly unhelpful error message :smiley: The "in" operator can be used for a sub-string match, or to test array membership. You cannot use it to test if one array is part of another. You can reproduce the error using

input { generator { count => 1 lines => [ '' ] tags => [ "abc" ] } }
filter {
    if [ "vpn" ] in [tags] { mutate { add_field => { "matched" => true } } }
}
output { stdout { codec => rubydebug { metadata => false } } }

Change if [ "vpn" ] in [tags] to if "vpn" in [tags]

@Badger , thanks alot. Can't believe such a stupid mistake. That's why peer code review is important :grin:.
Well it resolved the NullPointerException error. However now I've to deal with dissect error that is coming up. But I guess that I'll eventually figure out. Currently caught up with something else.
Thanks for the help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.