Custom filters are not working with Logstash-5.0.0

Elastic Stack Logstash
#1

I am new for logstash 5.0 and tried to create custom filter at 'PATH/logstash/filters/global_config.rb' with the name of global config but I am getting error as:
bin/logstash -f conf_d/
Sending Logstash logs to /usr/local/seceon/ashish/logstash-5.0.0/logs which is now configured via log4j2.properties.
[2016-11-04T15:41:44,592][ERROR][logstash.agent ] fetched an invalid config {:config=>"input { stdin { } }\n\nfilter {\n grok {\n match => { "message" => "%{COMBINEDAPACHELOG}" }\n }\n date {\n match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]\n }\n global_config {\n }\n}\n\noutput {\n elasticsearch { hosts => ["localhost:9200"] }\n stdout { codec => rubydebug }\n}\n\n", :reason=>"Couldn't find any filter plugin named 'global_config'. Are you sure this is correct? Trying to load the global_config filter plugin resulted in this error: LoadError"}

I don't know what's I am doing wrong here can someone please help me ?

Same global-config.rb is working fine with logstash-2.4.0

#2

I tried to copy mutate.rb and renamed it as mutate_1.rb getting this error

[2016-11-04T16:32:53,500][DEBUG][logstash.plugin ] Problems loading the plugin with {:type=>"filter", :name=>"mutate_1", :path=>"logstash/filters/mutate_1"}
[2016-11-04T16:32:53,509][ERROR][logstash.agent ] fetched an invalid config {:config=>"input { stdin { } }\n\nfilter {\n grok {\n match => { "message" => "%{COMBINEDAPACHELOG}" }\n }\n date {\n match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]\n }\n mutate_1 {\n }\n}\n\noutput {\n elasticsearch { hosts => ["localhost:9200"] }\n stdout { codec => rubydebug }\n}\n\n", :reason=>"Couldn't find any filter plugin named 'mutate_1'. Are you sure this is correct? Trying to load the mutate_1 filter plugin resulted in this error: LoadError", :backtrace=>["/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/plugin.rb:127:in lookup'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:413:inplugin'", "(eval):64:in initialize'", "org/jruby/RubyKernel.java:1079:ineval'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/agent.rb:195:increate_pipeline'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/agent.rb:87:in register_pipeline'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/runner.rb:258:inexecute'", "/usr/local/seceon/ashish/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/runner.rb:174:inrun'", "/usr/local/seceon/ashish/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/local/seceon/ashish/logstash-5.0.0/lib/bootstrap/environment.rb:68:in(root)'"]}

#3

You have an odd number of brackets in the error report. So check your config file and make sure you have properly placed the brackets correctly and didn't forget one :slight_smile: . Also if you can post the config file it will help.

#4

Thanks for reply..

I have a very small config with input/filter/output .
This is the config file
[root@cce-logs-processor logstash-5.0.0]# cat conf_d/test.conf
input { stdin { } }

filter {
global_config {
}
}

output {
stdout { codec => rubydebug }
}

and this is the error :slight_smile:
[2016-11-07T11:06:34,665][DEBUG][logstash.agent ] Reading config file {:config_file=>"/usr/local/seceon/ashish/logstash-5.0.0/conf_d/test.conf"}
it's here logstash/inputs/stdin
it's here stdin
it's here logstash/codecs/line
it's here line
[2016-11-07T11:06:34,690][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@id = "line_b0144465-2a1e-42ac-a98d-6e521933dfda"
[2016-11-07T11:06:34,690][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@enable_metric = true
[2016-11-07T11:06:34,690][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@charset = "UTF-8"
[2016-11-07T11:06:34,690][DEBUG][logstash.codecs.line ] config LogStash::Codecs::Line/@delimiter = "\n"
[2016-11-07T11:06:34,692][DEBUG][logstash.inputs.stdin ] config LogStash::Inputs::Stdin/@id = "a2cbc585fb561768566d789e197abb1de72fe3cb-1"
[2016-11-07T11:06:34,692][DEBUG][logstash.inputs.stdin ] config LogStash::Inputs::Stdin/@enable_metric = true
[2016-11-07T11:06:34,692][DEBUG][logstash.inputs.stdin ] config LogStash::Inputs::Stdin/@codec = <LogStash::Codecs::Line id=>"line_b0144465-2a1e-42ac-a98d-6e521933dfda", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">
[2016-11-07T11:06:34,692][DEBUG][logstash.inputs.stdin ] config LogStash::Inputs::Stdin/@add_field = {}
[2016-11-07T11:06:34,693][DEBUG][logstash.plugin ] Problems loading the plugin with {:type=>"filter", :name=>"global_config", :path=>"logstash/filters/global_config"}
[2016-11-07T11:06:34,699][ERROR][logstash.agent ] fetched an invalid config {:config=>"input { stdin { } }\n\nfilter {\n global_config {\n }\n}\n\noutput {\n stdout { codec => rubydebug }\n}\n\n", :reason=>"Couldn't find any filter plugin named 'global_config'. Are you sure this is correct? Trying to load the global_config filter plugin resulted in this error: LoadError", :backtrace=>["/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/plugin.rb:132:in lookup'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:413:inplugin'", "(eval):12:in initialize'", "org/jruby/RubyKernel.java:1079:ineval'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/agent.rb:195:increate_pipeline'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/agent.rb:87:in register_pipeline'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/runner.rb:258:inexecute'", "/usr/local/seceon/ashish/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/local/seceon/ashish/logstash-5.0.0/logstash-core/lib/logstash/runner.rb:174:inrun'", "/usr/local/seceon/ashish/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/local/seceon/ashish/logstash-5.0.0/lib/bootstrap/environment.rb:68:in(root)'"]}

#5

I see the same issue with my custom output plugin.
Attempt to start logstash 5.0 with a custom output plugin (Ensured that I ve configured path.plugins correctly in the logstash.yml ) fails with the following error -

Trying to load the XXXXX output plugin resulted in this error: LoadError

I feel that for some reasons the register method in the plugin is not getting invoked. I debugged and see that the exception is being thrown at:
registry.rb:69:in `lookup'

which means that the plugin registration indeed did not happen. And hence there is no entry for path,klass for my custom o/p plugin.
This was working all fine until logstash 2.2.1.

I browsed through the documentation for breaking changes, and followed whatever is being asked for. However, still I see this error.
Please note: I am not using plugin-install. I am trying to use the output plugin in a standalone fashion as a java based custom plugin.

Any help on this would be appreciated.
Thanks!
Yogesh

#6

Hi Yogesh ,
Did you find a solution for your issue ?

#7

Nothing yet Ashish.
Actually, I haven't put more efforts on this after this post, as I think I am blocked on this.

Can some logstash SMEs please help us out here?

#8

Is this an Issue with Logstash 5.0?
If so, when can we see this fixed? Any probable date /next version in which this can be seen fixed?

#9