I have solved the issue. Mostly thanks to this blog post you wrote:
Specifically, I have enabled eager_global_ordinals on the field and lengthening the refresh interval. It is now working very well. If I run into issues again i will update this issue.
Thanks — I know that this is mostly me adding updates to this topic, but I couldn’t have figured this out if not for that blog post!