[Need Help] Logstash start failed when use 3rd party gem in customized plugin

Elastic Stack Logstash
1

I used mail gem in customized codec plugin to handle some email related data, the plugin gem run bundle exec rspec and gem build is successful. The problem is when the plugin installed, it will lead Logstash start process failed.I tried to add --debug option to check the debug log when starting, it's weird:

The error reported is:
  Couldn't find any output plugin named 'stdout'. Are you sure this is correct? Trying to load the stdout output plugin resulted in this error: undefined method `on_load' for ActiveSupport:Module
/Users/lipy/elk/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/plugin.rb:142:in `lookup'
/Users/lipy/elk/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:293:in `plugin'
(eval):59:in `initialize'
org/jruby/RubyKernel.java:1111:in `eval'
/Users/lipy/elk/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:31:in `initialize'
/Users/lipy/elk/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/agent.rb:114:in `execute'
/Users/lipy/elk/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:90:in `run'
org/jruby/RubyProc.java:271:in `call'
/Users/lipy/elk/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/runner.rb:95:in `run'
org/jruby/RubyProc.java:271:in `call'
/Users/lipy/elk/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/task.rb:12:in `initialize'

But stdout plugin is fine, OK?!

I does a lot of check and found perhaps the root cause is mail gem, because there is only one line code related to mail gem in my plugin's source code, just require "mail", when I comment out this code line and rebuild gem and re-install plugin, the Logstash start successful. I checked the vendor folder of Logstash, both mail gem and the customized codec plugin has exist.

I'm stuck!! Help, please!!

There is some code for reference:

My ruby version:

jruby 1.7.19 (1.9.3p551) 2015-01-29 20786bd on Java HotSpot(TM) 64-Bit Server VM 1.8.0_40-b26 +jit [darwin-x86_64]

Gemfile:

source 'https://rubygems.org'
gemspec

gemsepc:

Gem::Specification.new do |s|
  # Leave out some general gem info
  ........
  # Files
  s.files           = `git ls-files`.split($\)

  # Tests
  s.test_files      = s.files.grep(%r{^(test|spec|features)/})

  # Special flag to let us know this is actually a logstash plugin
  s.metadata        = { 'logstash_plugin' => 'true', 'logstash_group' => 'codec' }

  # Gem dependencies
  s.add_runtime_dependency 'logstash-core', '>= 1.4.0', '< 2.0.0'
  s.add_runtime_dependency 'mail', '~>2.6', '>=2.6.3'
  s.add_development_dependency 'logstash-devutils', '~>0'
end
2

It might be helpful if we could review your configuration, and perhaps the code of your plugin (though we understand that you may be sensitive to sharing your private code).

3

Hi theuntergeek,

I'm happy to share my code, there is no secret :smile:

https://github.com/ttys000/logstash-codec-emailsubject, this is the plugin code.

My Logstash version is 1.5.2, here is the config file: https://gist.github.com/ttys000/439a20cde5662edd31b0

You need to import grok-patterns from this repo https://github.com/logstash-plugins/logstash-patterns-core

Thanks!

4

As an unrelated aside:

    "size" => "integer"
    "nrcpts" => "integer"
    "number_attachments" => "integer"
    "spamscore" => "integer"
    "spamthreshold" => "integer"
    "iascore" => "integer"
    "iathreshold" => "integer"
    "dlpfilesize" => "integer"
    "ts_reputation_score" => "integer"

Since you're not acting on any of these numerics inside Logstash, you can improve Logstash performance by coercing the types within an Elasticsearch template. Use the smallest numeric type available in Elasticsearch that suits the number. If the numbers are from 0-100, then a byte mapping even makes sense, otherwise a short might be better.

5

What's in the Gemfile.lock in your directory?

6

Thanks advice, I will check it out.

I uploaded the Gemfile.lock file into Github, you can pull it.

7

How have you installed Logstash?

8

I just unpack the .tar.gz file and run sudo ./logstash -f myconfig.conf &
What's matter?

9

How did you install your plugin into that installation?

10

Use this instruction:
./plugin install logstash-codec-emailsubject-0.1.0.gem