Repetitive JDBC input runs

Elastic Stack Logstash
1

We have a stored procedure which returns data for the last hour when called without any parameters. We scheduled the JDBC input plugin to run every hour and it works well. Questions is, if I want to ingest older data I can give a specific date/time to the stored procedure, but what's the best way to do that 300 times?
I can keep updating config file and restarting logstash, but this would be a pain to automate. Has anyone figured out a better way to take care of similar issue?

Thanks!

2

So the stored procedure always returns rows for a particular hour, but you can supply a parameter to select which hour to return?

3

Yes. Something like:

call getmedata(NULL); # Current hour
call getmedata("0801140000") # 2:00PM on Aug 1st
4

I don't see a clean way of doing that. Apart from updating the config file and restarting Logstash you could store the timestamp in an environment variable that you reference in your configuration. That might be slightly easier to automate.

5

I ended up updating config and restarting Logstash in a loop. Plugins like that desperately need dynamic variables. This way just feels wrong even though it works. Thanks for your help.

6

If you start logstash with -r then it will re-read the configuration when it changes. Saves you the huge overhead of restarting logstash for each config change.

7

But how would I know when the input completes? My loop simply waits for logstash to exit (after its done reading sql data), updates config and starts logstash again.

I can see how -r could be really helpful in a different situation though.

Thanks.

8

I do not have a good answer for that.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.