We are looking into ways to get some data from Salesforce into Elasticsearch. Mainly to help correlate customer support and sales events with our various log data.
However I am having trouble determining how often the Input goes and runs a SOQL query.
The documentation says:
This input plugin will stop after all the results of the query are processed and will
need to be re-run to fetch new results. It does not utilize the streaming API.
So how exactly do you re-run the input to fetch new results? Restarting Logstash? We were thinking of having it run frequently (every 1-5 minutes) so that we could reference the data in close to real time. The Logstash machines currently ingest a few thousand events per second from other sources so restarting isn't really an option.