We have been running 7.3 for a month now, and looking at ILM explain results, it is seen that the indices were getting rolled over at 50GB everytime. Even the last index, metricbeat-7.3.0-0000050 rolled over correctly from 49. The 51st rollover also worked as expected.
I can't do a dry run on that index now as I had to delete it to get the nodes started. But given that previous rollovers were successful, it might not be a problem with rollover itself?
After deleting the old 51 Index, I created a new 51 index to give a write index to the metricbeat-7.3.0 alias and continued indexing data. Now, I do see that 52nd rollover was successfully done today morning.
I did see the same problem with an APM (apm-7.3.0-span-0000029) index previously, and hence edited the default lifecycle policy packaged with beats to do a rollover on number of docs as well (default policy rolls over on 50gb) , because I believed that would've been the problem at that time.
However, I'll get the ES cluster upgraded to the 7.3.2 patch so that nodes don't crash again.