I need to lookup fields in a RDBMS (MySQL) during filter phase of Logstash to extract new informations.
Neverthless jdbc plugin exists only for input section.
I can't do this in input section because my fields are available only from the filter phase.
How can I do?
Thank you very much!
A few options come to mind:
- The translate filter might work, although tou won't be able to get real-time lookups.
- If you push the data from the relational database into an ES index you can use the elasticsearch filter.
- It should be possible to write a ruby filter that performs the lookups via e.g. HTTP.
Indeed jdbc filtering is not supported in current logstash plugin set , but there is existing work on github:
The last two are my experiments but it seems a few people are using it happily.
If it would suits your need, can you tell me which version of logstash you are currently using,
I would try to find some to publish the gem
1 Like
Thank you for your help magnusbaeck!
I've temporarily resolved with translate filter because data are small.
In next future I believe that I will write a ruby filter.
Thank you wiibaa.
I've been using logstash 2.0.0 and your plugin is very interesting but I'm not successfull to install it
because I'm newbie to ruby and I've some problems.
Hello again,
I have made the necessary for easy install of my plugin,
so you could try from your logstash installation to
bin/plugin install logstash-filter-jdbc_mysql
and add a filter configuration section in your logstash config.
If needed you can look at this example https://github.com/wiibaa/logstash-filter-jdbc_mysql/blob/master/lib/logstash/filters/jdbc_mysql.rb#L15-L25
Hello wiibaa,
thank you for your support!
I've tried to install your plugin but I've received this error
Validating logstash-filter-jdbc_mysql
SocketError: recv: name or service not known
recv_reply at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:782
request at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:664
each_resource at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:512
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:0
each at org/jruby/RubyArray.java:1613
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:1034
each at org/jruby/RubyArray.java:1613
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:1033
each at org/jruby/RubyArray.java:1613
resolv at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:1031
each_resource at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:503
getresource at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/1.9/resolv.rb:480
api_endpoint at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/remote_fetcher.rb:92
api_uri at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/source.rb:46
load_specs at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/source.rb:182
tuples_for at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:261
available_specs at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:226
each at org/jruby/RubyArray.java:1613
each_source at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/source_list.rb:97
available_specs at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:222
search_for_dependency at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:102
spec_for_dependency at /home/alessandro/Scrivania/logstash-2.0.0/vendor/jruby/lib/ruby/shared/rubygems/spec_fetcher.rb:166
logstash_plugin? at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/util.rb:21
verify_remote! at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/install.rb:50
each at org/jruby/RubyArray.java:1613
verify_remote! at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/install.rb:48
execute at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/install.rb:26
run at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
execute at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
run at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
run at /home/alessandro/Scrivania/logstash-2.0.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
(root) at /home/alessandro/Scrivania/logstash-2.0.0/lib/pluginmanager/main.rb:37
This strongly seems like a transient error with rubygems, not something related to the plugin or logstash.
Or are you maybe on an environment with restricted/proxied access to the internet ?