We've taken a few more heap dumps over the last 24 hours. One thing is beginning to stand out.
The taskmanager map keeps growing. It has 1.5 million instances of ReplicationTask.
We've looked into a few of the task objects and we can see they have phase = finished.
Seems like maybe the finished tasks are not being removed, or maybe the tasks are somehow getting stuck in a weird state, and therefore not being reaped.
We will keep taking heap dumps, but so far this seems a likely culprit.
cheers