I've been battling with http_poller and exec and pipe and can not seem to find the best method to accomplish this below input.
I have a API that only returns 100 results for each call and I need to bring in records for the past 10 years.
I have a start_date and end_date in my parameters.
What I have tried.
I created multiple http_poller lines for each date range that resulted in less then the 100 records which required having well over 100 inputs for the CURL's for each range. Worked but messy and you have to update the config manually for each new range.
For exec and pipe inputs
I created a BASH script that used a While Do that would loop thru the date ranges one day at a time which with my setup would result in less then 100 records a day and was safe. Problem was that only the last CURL call would do into my INPUT none of the results from the previous date ranges would input. The script works in Terminal and outputs fine to terminal. I think it's because Logstash waits for the final Event to end and it takes when ever was generated from that last CURL call.
Now I have my script running with out a While Do and it runs multiple times using the interval and in order for the variables to increment I have a File that I am grabbing the variable from. Problem was I could not echo the next +1 variable to that file in the same script as my CURL call. I couldn't even echo the var +1 to a file in another exec input since it was needing a while do. (does Logstash not run scripts the same as if they are ran in terminal. Because as a solution I now have the while do script that increments the file var running outside of Logstash in terminal)???
If there is a way to make the above logic work or if there is a way that is better to deal with this situation I am looking for any input that will make this process more contained and manageable.