Installing a custom local plugin

Elastic Stack Logstash
1

Hi all,

I want to make a bit modification to http input plugin, so to start with this task, i following steps:

  1. Install logstash 7.14 downloading logstash-7.14.0-darwin-x86_64.tar.gz, and test if installation is ok
bin/logstash -e 'input{stdin{}}'
  1. Clone the input plugin:
git https://github.com/logstash-plugins/logstash-input-http.git
  1. Build input plugin without any modification.
[gustavo@MacBookGus logstash-input-http (master)]$ gem build logstash-input-http.gemspec
WARNING:  license value 'Apache License (2.0)' is invalid.  Use a license identifier from
http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
Did you mean 'Apache-2.0'?
WARNING:  open-ended dependency on logstash-codec-plain (>= 0) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on logstash-devutils (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on logstash-codec-json (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on logstash-codec-json_lines (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on manticore (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: logstash-input-http
  Version: 3.4.1
  File: logstash-input-http-3.4.1-java.gem
  1. Remove current http input plugin included in logstash installataion.
[gustavo@MacBookGus logstash-7.14.0 ]$ bin/logstash-plugin remove logstash-input-http
Using bundled JDK: /Users/gustavo/Downloads/logstash-7.14.0/jdk.app/Contents/Home
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
/Users/gustavo/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Successfully removed logstash-input-http
  1. Install local plugin recently built
bin/logstash-plugin install /Users/gustavo/bold/devel/logstash-input-http/logstash-input-http-3.4.1-java.gem
Using bundled JDK: /Users/gustavo/Downloads/logstash-7.14.0/jdk.app/Contents/Home
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating /Users/gustavo/bold/devel/logstash-input-http/logstash-input-http-3.4.1-java.gem
Installing logstash-input-http
/Users/gustavo/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Installation successful
  1. Test input plugin recently installed.
[gustavo@MacBookGus logstash-7.14.0 ]$ bin/logstash -e 'input{http{}}'
Using bundled JDK: /Users/gustavo/Downloads/logstash-7.14.0/jdk.app/Contents/Home
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
/Users/gustavo/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Sending Logstash logs to /Users/gustavo/Downloads/logstash-7.14.0/logs which is now configured via log4j2.properties
[2021-08-05T14:57:49,860][INFO ][logstash.runner          ] Log4j configuration path used is: /Users/gustavo/Downloads/logstash-7.14.0/config/log4j2.properties
[2021-08-05T14:57:49,953][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.14.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [darwin-x86_64]"}
[2021-08-05T14:57:50,247][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-05T14:57:51,818][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-08-05T14:57:52,323][INFO ][org.reflections.Reflections] Reflections took 146 ms to scan 1 urls, producing 120 keys and 417 values
[2021-08-05T14:57:52,972][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"http"}
[2021-08-05T14:57:52,984][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'http'. Are you sure this is correct? Trying to load the http input plugin resulted in this error: Unable to load the requested plugin named http of type input. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/agent.rb:391)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:135
             add at org/logstash/execution/ConvergeResultExt.java:60
  converge_state at /Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/agent.rb:404
[2021-08-05T14:57:52,999][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2021-08-05T14:57:53,014][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/agent.rb:404:in `block in converge_state'"]}
[2021-08-05T14:57:53,044][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
	at Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.lib.bootstrap.environment.<main>(/Users/gustavo/Downloads/logstash-7.14.0/lib/bootstrap/environment.rb:89) ~[?:?]

I wonder if this error is due the warnings when i was build the gem.

I really appreciate your helps.

Thanks in advance.

Gustavo

2

Adding more info.
I running logstash again, with debug level log enabled,

bin/logstash --log.level debug -e 'input{http{}}'
..
[2021-08-05T15:10:14,422][DEBUG][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash-input-http_jars>, :path=>"logstash-input-http_jars", :type=>"input", :name=>"http"}
[2021-08-05T15:10:14,429][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"http"}
3

Finally i found the problem.

I missed build logstash-input-http plugin and i hadn't installed jruby in my system.

these are the steps i followed to resolve this:

  1. brew install jruby
  2. Clone logstash project and build it.
  3. On plugin project run
jruby -S bundle install
gradlew build
gradlew vendor
  1. Install this local plugin in my logstash installed and after that the jars missing was generated and the filter was found.

All work fine !

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.