How to get JDBC Data from Multiple Queries

Elastic Stack Logstash
1

Hello,

I'm trying to collect data from a DB (Derby, MySQL, Oracle) and my Schema is like this:

the main table is "exchange" and I need to get data from all other tables.
I'm trying to use logstash-input-jdbc, but with this input I can only configure one query and because the relations between the tables are 1->N, I can't collect all the data I need.

My goal is in the final to get something like this:

{
   ExchangeId => "",
   Role => "",
   EndPointService => "",
   ...
   exchangeproperty => [
                                     {
                                        ExchangeId => "",
                                        Name => "",
                                        Value => "",
                                     },
                                     { ... }, ...
                                   ]
  normalizemessage => [ ..... ]
}

At first, I thought to build a new Filter Plugin. My idea was, configure the logstash-input-jdbc plugin to Select the main table, and then with my new Filter Plugin, make all other queries with the ExchangeId return from Input.
When I try to implment my ideia, I follow this tutorial, but when I try to "bundle install" I'm getting the following error:

There was a NoMethodError while loading logstash-filter-jdbc.gemspec: 
undefined method `metadata=' for #<Gem::Specification name=logstash-filter-jdbc
version=0.0.1> from
/home/kintas/RubymineProjects/logstash-filter-jdbc/logstash-filter-jdbc.gemspec:18:in
`block in <main>'

I also try to build a new Input Plugin but I'm getting the same error.

So, what is the best way to accomplish my Goal? A new filter plugin, a new input plugin or other plugin that allready exists?

Thanks,
Kintas

2

would you mind sharing your gemspec for this filter?

I would take a look at some other plugin's gemspec and following that.

3

@talevy

I error I previous report was because my Ruby version was 1.9.1
After installing version 2.2.2p95 this error disapear.

But when I try 'bundle install' some gems couldn't be found.

"kintas@Kamaleon:~/RubymineProjects/logstash-filter-jdbc$ bundle install
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Could not find gem 'logstash-devutils (>= 0) ruby' in any of the gem sources
listed in your Gemfile or available on this machine.
"
you can get the gemfile in "https://github.com/GrupoFirst/logstash-input-jdbc"

thanks

4

JRuby is required for this plugin.

5

Hello @Kintas, I know it's an old thread, but you can achieve this with jdbc-streaming plugin.

I have a situation like yours, and I'm using jdbc-input to get the main data, and jdbc-streaming to enhance the message with more data.