What could be the cause of error ""

[Chen_Wei]

I recently met a error about timestamp, the error appears after the logstash pipeline start and work for a while. I could not reproduce it. But I wonder where the error could happen. Does it happen in the input plugin? since it says something about the codec.

Pipeline worker error
the pipeline will be stopped {:pipeline_id=>"main"
:error=>"(GeneratorError) Cannot invoke "org.logstash.Timestamp.toString()" because the return value of "org.logstash.ext.JrubyTimestampExtLibrary$RubyTimestamp.getTimestamp()" is null (through reference chain: org.logstash.ConvertedMap["_@timestamp"])"
:exception=>Java::OrgJrubyExceptions::Exception
:backtrace=>["RUBY.encode(/log_analyzer/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69)"
"RUBY.encode(/log_analyzer/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48)"
"org.logstash.instrument.metrics.AbstractSimpleMetricExt.time(org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74)"
"org.logstash.instrument.metrics.AbstractNamespacedMetricExt.time(org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68)"
"RUBY.encode(/log_analyzer/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47)"
"RUBY.multi_receive(/log_analyzer/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-integration-kafka-11.3.2-java/lib/logstash/outputs/kafka.rb:222)"
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1987)"
"RUBY.multi_receive(/log_analyzer/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-integration-kafka-11.3.2-java/lib/logstash/outputs/kafka.rb:221)"
"org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)"
"RUBY.start_workers(/log_analyzer/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"]
:thread=>"#<Thread:0xfd57d9a /log_analyzer/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}

[Badger]

This answer includes speculation....

The error in in the kafka output, which is calling a json codec. It is trying to encode a field. It seems likely that that field is called "_@timestamp". That field would get created if you have a json codec on an input and it receives JSON "@timestamp": "something that is not a valid time".

I am not inclined to work out what might provoke the code to return null, nor do I understand the message about the reference chain.

The underlying error is probably that a json codec on an input is failing to parse a field called @timestamp.