Logstash plugin crashing when calling .jar

Elastic Stack Logstash
1

I am currently running Logstash 5.6.7, due to an older version of Elastic Stack being provided by our DBA (we don't expect to have an upgrade until next year).

I've created a logstash plugin in jruby, which calls a jar file in order to get data from a REST API.
I run the plugin as follows:

gem build logstash-filter-example.gemspec &&
../../logstash-5.6.7.freshinstall/bin/logstash-plugin install logstash-filter-example-0.1.1.gem &&
../../logstash-5.6.7.freshinstall/bin/logstash -e 'input { stdin{} } filter { example {} } output {stdout { codec => rubydebug }}'

Upon running, the plugin suddenly fails when the jar file tries to convert a bigint (happens to be a negative number) into a JSON string. Strangely enough, the error is not reproducible when calling the jar from other java programs or jruby programs, only in Logstash.

Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x7dd1fccd @metric_events_out=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"52a1b3f6752b2c139814e0d27d8c61b7dc5e3121-2\", :events] key: out value: 0, @metric_events_in=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"52a1b3f6752b2c139814e0d27d8c61b7dc5e3121-2\", :events] key: in value: 0, @logger=#<LogStash::Logging::Logger:0x772997bd @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0x46efba22>>, @metric_events_time=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"52a1b3f6752b2c139814e0d27d8c61b7dc5e3121-2\", :events] key: duration_in_millis value: 0, @id=\"52a1b3f6752b2c139814e0d27d8c61b7dc5e3121-2\", @klass=LogStash::Filters::Example, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x406eb220 @metric=#<LogStash::Instrument::Metric:0x353610f @collector=#<LogStash::Instrument::Collector:0x38fc9aa7 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x5204e873 @store=#<Concurrent::Map:0x00000000066f34 entries=2 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x67cebfa1>, @fast_lookup=#<Concurrent::Map:0x00000000066f38 entries=57 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"52a1b3f6752b2c139814e0d27d8c61b7dc5e3121-2\", :events]>, @filter=<LogStash::Filters::Example id=>\"52a1b3f6752b2c139814e0d27d8c61b7dc5e3121-2\", enable_metric=>true, periodic_flush=>false>>", :error=>"LogStash::Filters::Example#register must be overidden"}

Pipeline aborted due to error {:exception=>#<RuntimeError: LogStash::Filters::Example#register must be overidden>, :backtrace=>["/Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filters/base.rb:134:in register'", "/Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:290:in register_plugin'", "/Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "org/jruby/RubyArray.java:1613:in each'", "/Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "/Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:311:in start_workers'", "/Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:235:in run'", "/Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/agent.rb:408:in start_pipeline'"]}

Does Logstash have any known compatibility issues related to the case above?

2

When extending LogStash::Filters::Base you must override both the register and filter methods.

3

Currently both of these methods are being overridden.
I originally created my file as described in the below link:

https://www.elastic.co/guide/en/logstash/5.6/_how_to_write_a_logstash_filter_plugin.html

This worked with no problem as a basic hello world type application.
Once this worked, I only made some small modifications to call my code from the jar file I created:

# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "java"
$LOAD_PATH << './lib'
require "sample.project.main.jar"

java_import "com.mike.dataservice.DataServiceClient"

# This  filter will replace the contents of the default 
# message field with whatever you specify in the configuration.
#
# It is only intended to be used as an .
class LogStash::Filters::Example < LogStash::Filters::Base

  config_name "example"
  config :message, :validate => :string, :default => DataServiceClient.new.getData   
  
  public
  def register
    # Add instance variables 
  end # def register

  public
  def filter(event)

    if @message
      # Replace the event message with our message as configured in the
      # config file.
      event.set("message", @message)
    end

    # filter_matched should go in the last line of our successful code
    filter_matched(event)
  end # def filter
end # class LogStash::Filters::Example
4

I moved the DataServiceClient.new.getData to the filter method and now a stack trace appears when the jar fails:

/dataservice/DataServiceClient.java:156:in `parseQueryRowEntry': java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.JsonNodeFactory.numberNode(Ljava/math/BigDecimal;)Lcom/fasterxml/jackson/databind/node/NumericNode;
	from com/mike/dataservice/DataServiceClient.java:93:in `toJsonObject'
	from com/mike/dataservice/DataServiceClient.java:66:in `getData'
	from java/lang/reflect/Method.java:498:in `invoke'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/vendor/local_gems/ecdbfdf7/logstash-filter-example-0.1.1/lib/logstash/filters/example.rb:49:in `filter'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'
	from org/jruby/RubyArray.java:1613:in `each'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filter_delegator.rb:46:in `multi_filter'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/filter_delegator.rb:46:in `multi_filter'
	from (eval):42:in `filter_func'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:398:in `filter_batch'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:379:in `worker_loop'
	from /Users/michael.dobrin/compilationsFromSource/logstash-5.6.7/logstash-5.6.7.freshinstall/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'
	from java/lang/Thread.java:748:in `run'

This works fine when running the jar directly from other sources.
The parseQueryRowEntry() method certainly exists when I build, but for some reason, when I run Logstash the method no longer exists.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.