Thanks so much for your detailed reply @_markus! It sounds like you've tracked it down, and have saved me from spending lots of time digging into things. It seems like something that would be good to have called out in the docs - maybe even just for the refresh=wait_for setting to let you know that it will take a long time.
For our implementation we have parent child relationships but are in the process of migrating away from them, we already have stopped querying them and are about to remove them from our index completely. I'll let you know how it goes!