Is there a method to disable using the scroll_id on the Logstash Input Plugin for Elasticsearch? It appears that because the query uses a scroll_id, it causes data to be missed (always viewing a snap shot).
I am using an ES Index as Input, processing, and using a new ES index for Output. However, as new documents are inserted into the input Index, they are not detected by Logstash. If I restart Logstash, it will collect all events that were missed.
I think it is because underneath each Input query is using the previous _scroll_id. I need to just use a simple query, to get the same results each time.
I've tried a few of the options hoping it would disable the scroll, but to no avail.