Installing a custom local plugin

gusDuarte · August 5, 2021, 6:03pm

Hi all,

I want to make a bit modification to http input plugin, so to start with this task, i following steps:

Install logstash 7.14 downloading logstash-7.14.0-darwin-x86_64.tar.gz, and test if installation is ok

bin/logstash -e 'input{stdin{}}'

Clone the input plugin:

git https://github.com/logstash-plugins/logstash-input-http.git

Build input plugin without any modification.

[gustavo@MacBookGus logstash-input-http (master)]$ gem build logstash-input-http.gemspec
WARNING:  license value 'Apache License (2.0)' is invalid.  Use a license identifier from
http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
Did you mean 'Apache-2.0'?
WARNING:  open-ended dependency on logstash-codec-plain (>= 0) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on logstash-devutils (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on logstash-codec-json (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on logstash-codec-json_lines (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on manticore (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: logstash-input-http
  Version: 3.4.1
  File: logstash-input-http-3.4.1-java.gem

Remove current http input plugin included in logstash installataion.

[gustavo@MacBookGus logstash-7.14.0 ]$ bin/logstash-plugin remove logstash-input-http
Using bundled JDK: /Users/gustavo/Downloads/logstash-7.14.0/jdk.app/Contents/Home
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
/Users/gustavo/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Successfully removed logstash-input-http

Install local plugin recently built

bin/logstash-plugin install /Users/gustavo/bold/devel/logstash-input-http/logstash-input-http-3.4.1-java.gem
Using bundled JDK: /Users/gustavo/Downloads/logstash-7.14.0/jdk.app/Contents/Home
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating /Users/gustavo/bold/devel/logstash-input-http/logstash-input-http-3.4.1-java.gem
Installing logstash-input-http
/Users/gustavo/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Installation successful

Test input plugin recently installed.

[gustavo@MacBookGus logstash-7.14.0 ]$ bin/logstash -e 'input{http{}}'
Using bundled JDK: /Users/gustavo/Downloads/logstash-7.14.0/jdk.app/Contents/Home
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
/Users/gustavo/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated
Sending Logstash logs to /Users/gustavo/Downloads/logstash-7.14.0/logs which is now configured via log4j2.properties
[2021-08-05T14:57:49,860][INFO ][logstash.runner          ] Log4j configuration path used is: /Users/gustavo/Downloads/logstash-7.14.0/config/log4j2.properties
[2021-08-05T14:57:49,953][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.14.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [darwin-x86_64]"}
[2021-08-05T14:57:50,247][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-05T14:57:51,818][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-08-05T14:57:52,323][INFO ][org.reflections.Reflections] Reflections took 146 ms to scan 1 urls, producing 120 keys and 417 values
[2021-08-05T14:57:52,972][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"http"}
[2021-08-05T14:57:52,984][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'http'. Are you sure this is correct? Trying to load the http input plugin resulted in this error: Unable to load the requested plugin named http of type input. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/agent.rb:391)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:135
             add at org/logstash/execution/ConvergeResultExt.java:60
  converge_state at /Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/agent.rb:404
[2021-08-05T14:57:52,999][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2021-08-05T14:57:53,014][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/Users/gustavo/Downloads/logstash-7.14.0/logstash-core/lib/logstash/agent.rb:404:in `block in converge_state'"]}
[2021-08-05T14:57:53,044][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
	at Users.gustavo.Downloads.logstash_minus_7_dot_14_dot_0.lib.bootstrap.environment.<main>(/Users/gustavo/Downloads/logstash-7.14.0/lib/bootstrap/environment.rb:89) ~[?:?]

I wonder if this error is due the warnings when i was build the gem.

I really appreciate your helps.

Thanks in advance.

Gustavo

gusDuarte · August 5, 2021, 6:16pm

Adding more info.
I running logstash again, with debug level log enabled,

bin/logstash --log.level debug -e 'input{http{}}'
..
[2021-08-05T15:10:14,422][DEBUG][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash-input-http_jars>, :path=>"logstash-input-http_jars", :type=>"input", :name=>"http"}
[2021-08-05T15:10:14,429][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"http"}

gusDuarte · August 5, 2021, 8:43pm

Finally i found the problem.

I missed build logstash-input-http plugin and i hadn't installed jruby in my system.

these are the steps i followed to resolve this:

brew install jruby
Clone logstash project and build it.
On plugin project run

jruby -S bundle install
gradlew build
gradlew vendor

Install this local plugin in my logstash installed and after that the jars missing was generated and the filter was found.

All work fine !

system · September 2, 2021, 8:44pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.