In logstash can we have multiple inputs and combine the result set and process the combined data to one output?

Elastic Stack Logstash
1

Hi ,
I am using a config file like below. In the input section I am pulling the data from different database. I want to perform join and combine results in filter section. basically I just want to merge the result set ,after that the combined data should go to one output .
input
{
jdbc
{
//select * from db1.table1

}

jdbc
{
//select * from db2.table1
}

}

filter
{
}

output
{
// single output.
}

2

Logstash handles streams of events, so I do not think you can combine the data from the two inputs. You may however be able to use the jdbc streaming or jdbc static plugins to enrich the data generated through one of the inputs with data from the other database table.

3

as I can see we can give one jdbc_connection_string and deals with multiple tables. Is there any way we can give multiple connection string?

4

can we give jdbc_connection_string in the loaders section?

5

Can you explain what the data represents, where it resides and how you are looking to merge them?

6

lets say table1 from database1 has 100 rows having columns id and name. Similarly table2 from database2 has 100 rows having columns id, and city. I just want to perform join on both the tables and apply some filters on the result set, I can do this in sql file but I just want to know how logstash will behave to this scenario.

7

If you can, I would recommend performing the join in the database.

8

sure. But do we have connection string option in the below section of loader of jdbc static plug in.

loaders => [
{
id => "remote-servers"
query => "select ip, descr from ref.local_ips order by ip"
local_table => "servers"
},

9

If you are performing the join in the database prior to the data being read into Logstash through the input plugin, you will not need the plugins I linked to.

10

okay. thank you

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.