Hello,
I'm trying to figure out a way to have logstash run a sql query defined in the config file once and load all the data into ES.
If I run the query on the cron style schedule (which keeps logstash running) the data gets there just fine.
However, if I remove the schedule from the config file logstash only runs as long as the sql query is executing. It then exits, leaving most of the data in the persisted queue instead of sending it to ES.
I've tried enabling queue.drain hoping that logstash wouldn't exit until all items had been sent from the queue to ES, but that seems to make logstash freeze after the sql query is done executing. It doesn't keep sending data.
I'm using logstash 5.6.10. Upgrading to 6.x isn't possible at this time.
I really just want logstash to run the sql query and exit after it has sent all the data to ES. What am I missing?
Thank you.