[Solved] Problem using LDAP library in custom filter plugin

Elastic Stack Logstash
1

Hi everyone !

I'm running Logstash 2.1.2 inside a docker container. My configuration specifies to read data into a file and indexes data into an ES cluster.

I developped a custom filter plugin that queries a LDAP server for informations, but whenever it tries to call the library 'net/ldap', it fails with this error message :

"{:timestamp=>"2016-02-15T10:34:55.656000+0000", :message=>"Exception in filterworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.", "exception"=>#<LoadError: no such file to load -- net/ldap>, "backtrace"=>["org/jruby/RubyKernel.java:1040:in require'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:inrequire'", "/opt/logstash/vendor/local_gems/ab5428d7/logstash-filter-caparmor-0.0.1/lib/logstash/filters/caparmor.rb:872:in addUserInformations'", "/opt/logstash/vendor/local_gems/ab5428d7/logstash-filter-caparmor-0.0.1/lib/logstash/filters/caparmor.rb:223:inparse_message'", "/opt/logstash/vendor/local_gems/ab5428d7/logstash-filter-caparmor-0.0.1/lib/logstash/filters/caparmor.rb:41:in filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/filters/base.rb:151:inmulti_filter'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/filters/base.rb:148:inmulti_filter'", "(eval):1052:in cond_func_33'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):1049:in cond_func_33'", "(eval):496:infilter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/pipeline.rb:244:in filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/pipeline.rb:178:instart_filters'"], :level=>:error}
LoadError: no such file to load -- net/ldap
require at org/jruby/RubyKernel.java:1040
require at /opt/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65
addUserInformations at /opt/logstash/vendor/local_gems/ab5428d7/logstash-filter-caparmor-0.0.1/lib/logstash/filters/caparmor.rb:872
parse_message at /opt/logstash/vendor/local_gems/ab5428d7/logstash-filter-caparmor-0.0.1/lib/logstash/filters/caparmor.rb:223
filter at /opt/logstash/vendor/local_gems/ab5428d7/logstash-filter-caparmor-0.0.1/lib/logstash/filters/caparmor.rb:41
multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/filters/base.rb:151
each at org/jruby/RubyArray.java:1613
multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/filters/base.rb:148
cond_func_33 at (eval):1052
each at org/jruby/RubyArray.java:1613
cond_func_33 at (eval):1049
filter_func at (eval):496
filterworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/pipeline.rb:244
start_filters at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.2-java/lib/logstash/pipeline.rb:178
"

I did not forget to install net-ldap library with "gem install net-ldap".
My plugin also uses elasticsearch client library and it works well, so I don't really understand what's happening here.

This failure also happens if I try to use this plugin outside a docker container (with the latest version of LS).

When searching for this kind of problem, I met a topic that told to specify env variable GEM_HOME, but it did not help.

I'm pretty lost with this stuff, so I hope someone will be able to help :slightly_smiling:

2

I don't think running gem install in your shell is enough or a gem to be available when you run it in Logstash. I'd assume you'd have to specify that gem as a runtime dependency.

3

Hi, thanks for your answer.

Indeed, that was due to a bad way to declare dependency.

I managed to use net-ldap library thanks to this page

Two steps helped me to accomplish this :

  1. Installing the gem with this command : env GEM_HOME=/opt/logstash/vendor/bundle/jruby/1.9 /opt/logstash/vendor/jruby/bin/jruby /opt/logstash/vendor/jruby/bin/gem install net-ldap -v 0.12.0
  2. Adding gem "net-ldap", "0.12.0" in /opt/logstash/Gemfile

Works inside & outside docker containers ! :sunny: