Lumberjack with JDBC configuration

Elastic Stack Logstash
1

Hi,

Need help with a very typical scenario of getting data using JDBC input plugin and sending it using Logstash - Lumberjack(s) plugins to ES
Version Details: ES/LS/Kibana - 5.4.1

I have a database with 2 tables that i query separately to get some data
Query 1 - Get Events
Query 2 - Get Value

The setup is such as way that i have a server running logstash with JDBC input & Lumberjack output (forwarder)
CLIENT configuration is like

input{
jdbc{ ....Query 1}
}
output{
lumberjack { port => 5044
host => IPOFLUMBERJACKSERVER
}

Another server receives the data as Lumberjack input an sends it to ES cluster
SERVER configuration is like

input{
lumberjack{
port => 5044
}
output{
elasticsearch {
hosts => [ES_1:port, ES_2:port]
index => events 
 }

Now this configuration works absolutely fine if i have one query, I need to know how should i get these 2 queries parsed using single CLIENT configuration and get individual indices for different queries i.e "events" & "value"

2

You can e.g. set a field in each jdbc input (use type or add_field) and reference that field in your output configuration. This is a very common question so you should be able to find details and examples in the archives and/or Stackoverflow. The fact that you're shipping your events with the lumberjack plugins before it reaches the elasticsearch output is immaterial.

3

Thanks Magnus, I did tried that, the query 1 is started and created the index with all the events,
But the issue is my second query 2 is still waiting for query 1 to complete, is there a way to have get this sorted.
I have one database and 2 queries

4

That's surprising. I was under the impression that each jdbc input ran in its own thread.

5

Hi Magnus,
Tried running a schedule with other query (for events ) , seems the query 2 is also indexed in same "value" index but the type is generated as 111,124 etc and not "events"

IMG_1504
IMG_1504.JPG2933×3905 2.02 MB