Hey Elastic Community,
with the turn of May to June we had some issues with our Logstash.
To get Data from older documents we use the elasticsearch filter plugin in some of our pipelines. Most of our indices are month based named, so with month turnover a new index will be used.
Logstash Filter abstract:
input {
snmp {
...
}
}
filter {
elasticsearch {
hosts => ["..."]
query => "_id:%{portid}"
index => "xxx-%{+YYYY.MM}"
fields => {
...
}
}
}
output {
elasticsearch {
hosts => ["..."]
index => "xxx-%{+YYYY.MM}"
action => update
doc_as_upsert => true
document_id => "%{portid}"
}
elasticsearch {
hosts => ["..."]
index => "ed-interfacestats-%{+YYYY.MM}"
}
}
On the 1st of June Logstash throwed the following exception:
[2020-06-01T02:00:00,970][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"xxx-%{+YYYY.MM}", :error=>"[404] {\"error\":{\"root_cause\":
[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [xxx-2020.06]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"xxx-2020.06\",\"index_uuid\":\"
_na_\",\"index\":\"xxx-2020.06\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [xxx-2020.06]\",\"resource.type\":\"index_or_alias\",\"resource.id\":
\"xxx-2020.06\",\"index_uuid\":\"_na_\",\"index\":\"xxx-2020.06\"},\"status\":404}"}
[2020-06-01T02:00:00,971][ERROR][org.logstash.execution.WorkerLoop] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logsta
sh.
java.lang.NullPointerException: null
at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:659) ~[logstash-core.jar:?]
at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:433) ~[logstash-core.jar:?]
at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:424) ~[logstash-core.jar:?]
at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:27) ~[logstash-core.jar:?]
at org.logstash.generated.CompiledDataset226.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset238.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset253.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset259.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset260.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset261.compute(Unknown Source) ~[?:?]
at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64) [logstash-core.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor152.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441) [jruby-complete-9.2.9.0.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305) [jruby-complete-9.2.9.0.jar:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32) [jruby-complete-9.2.9.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:262) [jruby-complete-9.2.9.0.j
ar:?]
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.9.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.9.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71) [jruby-complete-9.2.9.0.jar:?]
at org.jruby.runtime.Block.call(Block.java:125) [jruby-complete-9.2.9.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.9.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.9.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-06-01T02:00:01,276][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.executi
on.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "jdk.internal.reflect.GeneratedMethodAccessor152.invoke(jdk/internal/reflect/GeneratedMethodAccessor152)", "jdk.internal.reflect.Delega
tingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.inv
okeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)", "usr.share.logstash.logstash_m
inus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:262)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "java.lang.Thread.ru
n(java/lang/Thread.java:834)"]}
[2020-06-01T02:00:01,601][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
I'm not sure, but in the past this did work without NP exception. The Logstash Service throwed a warning about the missing index and created the new index in the output section.
Is there any Elasticsearch Filter Plugin Parameter for Error Handling or did we observe a bug?
best regards
-Alufolie