How to get JDBC Data from Multiple Queries

(Pedro Quintas) #1


I'm trying to collect data from a DB (Derby, MySQL, Oracle) and my Schema is like this:

the main table is "exchange" and I need to get data from all other tables.
I'm trying to use logstash-input-jdbc, but with this input I can only configure one query and because the relations between the tables are 1->N, I can't collect all the data I need.

My goal is in the final to get something like this:

   ExchangeId => "",
   Role => "",
   EndPointService => "",
   exchangeproperty => [
                                        ExchangeId => "",
                                        Name => "",
                                        Value => "",
                                     { ... }, ...
  normalizemessage => [ ..... ]

At first, I thought to build a new Filter Plugin. My idea was, configure the logstash-input-jdbc plugin to Select the main table, and then with my new Filter Plugin, make all other queries with the ExchangeId return from Input.
When I try to implment my ideia, I follow this tutorial, but when I try to "bundle install" I'm getting the following error:

There was a NoMethodError while loading logstash-filter-jdbc.gemspec: 
undefined method `metadata=' for #<Gem::Specification name=logstash-filter-jdbc
version=0.0.1> from
`block in <main>'

I also try to build a new Input Plugin but I'm getting the same error.

So, what is the best way to accomplish my Goal? A new filter plugin, a new input plugin or other plugin that allready exists?


(Tal Levy) #2

would you mind sharing your gemspec for this filter?

I would take a look at some other plugin's gemspec and following that.

(Pedro Quintas) #3


I error I previous report was because my Ruby version was 1.9.1
After installing version 2.2.2p95 this error disapear.

But when I try 'bundle install' some gems couldn't be found.

"kintas@Kamaleon:~/RubymineProjects/logstash-filter-jdbc$ bundle install
Fetching gem metadata from
Fetching version metadata from
Fetching dependency metadata from
Could not find gem 'logstash-devutils (>= 0) ruby' in any of the gem sources
listed in your Gemfile or available on this machine.
you can get the gemfile in ""


(Tal Levy) #4

JRuby is required for this plugin.

(Enderson Maia) #5

Hello @Kintas, I know it's an old thread, but you can achieve this with jdbc-streaming plugin.

I have a situation like yours, and I'm using jdbc-input to get the main data, and jdbc-streaming to enhance the message with more data.

(system) #6