Custom Plugin not found

Elastic Stack Logstash
1

Im working this plugin to fetch logs from hdfs, this is the gemspec file:

  s.name          = 'logstash-input-webhdfs'
  s.version       = '0.1.0'
  s.licenses      = ['Apache License (2.0)']
  s.summary       = 'Write a short summary, because Rubygems requires one.'
  s.description   = 'Write a longer description or delete this line.'
  s.homepage      = 'http://localhost'
  s.authors       = ['xxxxxxx']
  s.email         = 'xxxxxxx@xxxxxx.xxx'
  s.require_paths = ['lib']

  # Files
  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
   # Tests
  s.test_files = s.files.grep(%r{^(test|spec|features)/})

  # Special flag to let us know this is actually a logstash plugin
  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }

  # Gem dependencies
  s.add_runtime_dependency 'logstash-core'
  s.add_runtime_dependency "logstash-core-plugin-api"
  s.add_runtime_dependency 'webhdfs'

  s.add_development_dependency 'logstash-devutils'
  s.add_development_dependency 'logstash-codec-line'

I create the gem file using gem build, then I install it with logstash-plugin install commad.
finally I check if the plugin is installed witth logstash-plugin list command.

When launching logstash this error popup :

[2017-03-22T11:15:41,977][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"input", :name=>"webhdfs", :path=>"logstash/inputs/webhdfs", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/opt/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:221:in `namespace_lookup'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:157:in `legacy_lookup'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:133:in `lookup'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:175:in `lookup_pipeline_plugin'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/plugin.rb:129:in `lookup'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:95:in `plugin'", "(eval):12:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:65:in `initialize'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:144:in `initialize'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/agent.rb:275:in `create_pipeline'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/runner.rb:264:in `execute'", "/opt/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/runner.rb:183:in `run'", "/opt/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/opt/logstash-5.2.2/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2017-03-22T11:15:41,979][DEBUG][logstash.plugins.registry] Problems loading the plugin with {:type=>"input", :name=>"webhdfs"}
[2017-03-22T11:15:41,985][ERROR][logstash.agent           ] Cannot load an invalid configuration {:reason=>"Couldn't find any input plugin named 'webhdfs'. Are you sure this is correct? Trying to load the webhdfs input plugin resulted in this error: Problems loading the requested plugin named webhdfs of type input. Error: NameError NameError", :backtrace=>["/opt/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/plugin.rb:129:in `lookup'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:95:in `plugin'", "(eval):12:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:65:in `initialize'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:144:in `initialize'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/agent.rb:275:in `create_pipeline'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/runner.rb:264:in `execute'", "/opt/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/opt/logstash-5.2.2/logstash-core/lib/logstash/runner.rb:183:in `run'", "/opt/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/opt/logstash-5.2.2/lib/bootstrap/environment.rb:71:in `(root)'"]}

The plugin is well listed, I tried to remove/install the plugin but i don't untersatand the resons.
Last, my logstash (5.2.2) is not running as a service.

Could you please help me to solve this. Tank you.

2

UpUp

3

make sure the config_name is set correctly in the plugin code, like in the example input plugin:

class LogStash::Inputs::Example < LogStash::Inputs::Base
  config_name "example"
  #....
end
4

Already done, but it's not the problem.

5

can you show the whole directory structure and, if possible, the code for the plugin?

6

Thank you for your answer.
my initial structure looks like this :

lib/logstash/inputs/webhdfs/webhdfs.rb
lib/logstash/inputs/webhdfs/webhdfs_connect.rb

my webhdfs_connect.rb conntains utility class used to connect to webhdfs and it was included in webhdfs.rb with

require_relative webhdfs_connect.rb

I managed to solve the problem by merging this two files in one big nasty file , I don't really understand how lib structure is interpreted by logstash-plugin so I can't explain why.