Search counts changing while refresh is disabled

Hello,
I've set my cluster configuration to have refresh_interval = -1 so I can ask for explicit refresh after completing my index update. This is to avoid numbers changing for users while making queries (count typed) until completing the update and calling refresh.
Unfortunately query results keep changing even though refresh is not being called at all.

Is there something I'm missing?
According to documentation, new documents (updated documents are also new in a way) should not be visible until refresh is being performed.

Thanks,
Oren