I figured out the reason for the issue I was facing. I was using the elasticsearch input plugin without a schedule attribute and hence, by pipeline ran once when I started/restarted logstash and then terminated after the run. I had to schedule it to run once per day (which was my requirement as well) using the schedule attribute and that made sure that my pipeline was in running state.
This resolved the issue I have posted about. When the aggregation event was generated 15 seconds after my pipeline ran each day, my pipeline is still in running state and the events go through to the output successfully.
Thank you