Lumberjack with JDBC configuration


Need help with a very typical scenario of getting data using JDBC input plugin and sending it using Logstash - Lumberjack(s) plugins to ES
Version Details: ES/LS/Kibana - 5.4.1

I have a database with 2 tables that i query separately to get some data
Query 1 - Get Events
Query 2 - Get Value

The setup is such as way that i have a server running logstash with JDBC input & Lumberjack output (forwarder)
CLIENT configuration is like

jdbc{ ....Query 1}
lumberjack { port => 5044

Another server receives the data as Lumberjack input an sends it to ES cluster
SERVER configuration is like

port => 5044
elasticsearch {
hosts => [ES_1:port, ES_2:port]
index => events 

Now this configuration works absolutely fine if i have one query, I need to know how should i get these 2 queries parsed using single CLIENT configuration and get individual indices for different queries i.e "events" & "value"

You can e.g. set a field in each jdbc input (use type or add_field) and reference that field in your output configuration. This is a very common question so you should be able to find details and examples in the archives and/or Stackoverflow. The fact that you're shipping your events with the lumberjack plugins before it reaches the elasticsearch output is immaterial.

Thanks Magnus, I did tried that, the query 1 is started and created the index with all the events,
But the issue is my second query 2 is still waiting for query 1 to complete, is there a way to have get this sorted.
I have one database and 2 queries

That's surprising. I was under the impression that each jdbc input ran in its own thread.

Hi Magnus,
Tried running a schedule with other query (for events ) , seems the query 2 is also indexed in same "value" index but the type is generated as 111,124 etc and not "events"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.