Dynamic Timestamp Pattern Failure for Date Filter

Elastic Stack Logstash
1

I'm trying to parse a date with a dynamic timestamp pattern provided by the client, but Logstash fails and won't start when accessing the timestamp_pattern variable in the date filter. Here is an example that reproduces the same error even when the timestamp_pattern variable is added as a field before executing the date filter:

input {
  stdin { }
}

filter {
  mutate {
    add_field => [ "timestamp_pattern", "EEE MMM dd HH:mm:ss yyyy SSS Z" ]
  }

  date {
    match => [ "timestamp", "%{[timestamp_pattern]}" ]
  }

}

output {
    stdout {
      codec => rubydebug { metadata => true }
    }
}

Error logs are included in the reply below due to post size limit.

2

Error Logs:

...
logstash    | [2019-10-31T20:55:12,391][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalArgumentException", :message=>"Illegal pattern component: t", :backtrace=>["org.joda.time.format.DateTimeFormat.parsePatternTo(org/joda/time/format/DateTimeFormat.java:566)", "org.joda.time.format.DateTimeFormat.createFormatterForPattern(org/joda/time/format/DateTimeFormat.java:687)", "org.joda.time.format.DateTimeFormat.forPattern(org/joda/time/format/DateTimeFormat.java:177)", "org.logstash.filters.parser.JodaParser.<init>(org/logstash/filters/parser/JodaParser.java:58)", "org.logstash.filters.parser.TimestampParserFactory.makeParser(org/logstash/filters/parser/TimestampParserFactory.java:60)", "org.logstash.filters.parser.TimestampParserFactory.makeParser(org/logstash/filters/parser/TimestampParserFactory.java:69)", "org.logstash.filters.DateFilter.acceptFilterConfig(org/logstash/filters/DateFilter.java:66)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:425)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:292)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_date_minus_3_dot_1_dot_9.lib.logstash.filters.date.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-date-3.1.9/lib/logstash/filters/date.rb:185)", "org.jruby.RubyArray.collect(org/jruby/RubyArray.java:2563)", "org.jruby.RubyArray.map(org/jruby/RubyArray.java:2577)", "org.jruby.RubyArray$INVOKER$i$0$0$map19.call(org/jruby/RubyArray$INVOKER$i$0$0$map19.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_date_minus_3_dot_1_dot_9.lib.logstash.filters.date.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-date-3.1.9/lib/logstash/filters/date.rb:184)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:894)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:798)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.plugins.PluginFactoryExt$Plugins.filter_delegator(org/logstash/plugins/PluginFactoryExt.java:78)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:248)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:184)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.plugin(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:71)", "RUBY.initialize((eval):93)", "org.jruby.RubyKernel.evalCommon(org/jruby/RubyKernel.java:1099)", "org.jruby.RubyKernel.eval(org/jruby/RubyKernel.java:1061)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval.call(org/jruby/RubyKernel$INVOKER$s$0$3$eval.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.initialize(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.initialize(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:266)", "org.jruby.RubyProc$INVOKER$i$0$0$call.call(org/jruby/RubyProc$INVOKER$i$0$0$call.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.exclusive(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:165)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(org/jruby/ext/thread/Mutex$INVOKER$i$0$0$synchronize.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.exclusive(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/agent.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/pipeline_action//usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
logstash    | warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
logstash    | LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalArgumentException` for `PipelineAction::Create<main>`
logstash    |           create at org/logstash/execution/ConvergeResultExt.java:109
logstash    |              add at org/logstash/execution/ConvergeResultExt.java:37
logstash    |   converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:347
logstash    | [2019-10-31T20:55:12,414][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalArgumentException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:347:in `block in converge_state'"]}
logstash    | [2019-10-31T20:55:12,520][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalArgumentException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:347:in `block in converge_state'"]}
...
logstash exited with code 1
3

I do not think you can use a sprintf reference in that context.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.