I think I may have found the problem. I still had the discover:sampleSize set.
The behavior looks like this: that setting isn't taken in account in the dashboard, unless you have the size field inside your saved object. So for the workaround to work for you, you need to set both limits: the size inside the the object and the advanced setting (which is the one whose number is taken into account).
I had added an issue on the Kibana repo and I'll update it with this new information. https://github.com/elastic/kibana/issues/11758