Jdbc in filter section


#1

I need to lookup fields in a RDBMS (MySQL) during filter phase of Logstash to extract new informations.
Neverthless jdbc plugin exists only for input section.
I can't do this in input section because my fields are available only from the filter phase.
How can I do?
Thank you very much!


(Magnus Bäck) #2

A few options come to mind:

  • The translate filter might work, although tou won't be able to get real-time lookups.
  • If you push the data from the relational database into an ES index you can use the elasticsearch filter.
  • It should be possible to write a ruby filter that performs the lookups via e.g. HTTP.

#3

Indeed jdbc filtering is not supported in current logstash plugin set , but there is existing work on github:



The last two are my experiments but it seems a few people are using it happily.
If it would suits your need, can you tell me which version of logstash you are currently using,
I would try to find some to publish the gem


#4

Thank you for your help magnusbaeck!
I've temporarily resolved with translate filter because data are small.
In next future I believe that I will write a ruby filter.


#5

Thank you wiibaa.
I've been using logstash 2.0.0 and your plugin is very interesting but I'm not successfull to install it
because I'm newbie to ruby and I've some problems.


#6

Hello again,

I have made the necessary for easy install of my plugin,
so you could try from your logstash installation to

bin/plugin install logstash-filter-jdbc_mysql

and add a filter configuration section in your logstash config.
If needed you can look at this example https://github.com/wiibaa/logstash-filter-jdbc_mysql/blob/master/lib/logstash/filters/jdbc_mysql.rb#L15-L25


#7

Hello wiibaa,
thank you for your support!
I've tried to install your plugin but I've received this error

Validating logstash-filter-jdbc_mysql
SocketError: recv: name or service not known
recv_reply at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:782
request at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:664
each_resource at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:512
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:0
each at org/jruby/RubyArray.java:1613
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:1034
each at org/jruby/RubyArray.java:1613
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:1033
each at org/jruby/RubyArray.java:1613
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:1031
each_resource at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:503
getresource at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:480
api_endpoint at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/remote_fetcher.rb:92
api_uri at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/source.rb:46
load_specs at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/source.rb:182
tuples_for at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:261
available_specs at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:226
each at org/jruby/RubyArray.java:1613
each_source at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/source_list.rb:97
available_specs at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:222
search_for_dependency at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:102
spec_for_dependency at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:166
logstash_plugin? at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/util.rb:21
verify_remote! at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/install.rb:50
each at org/jruby/RubyArray.java:1613
verify_remote! at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/install.rb:48
execute at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/install.rb:26
run at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
execute at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
run at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
run at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
(root) at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/main.rb:37


#8

This strongly seems like a transient error with rubygems, not something related to the plugin or logstash.
Or are you maybe on an environment with restricted/proxied access to the internet ?


(system) #9