I'm trying to make sense of this. It's strange because logging and file rotation are not related at all (or shouldn't be). Could it be a timing issue? Means if you have debug enabled, it is a little bit slower and to process events it takes longer so the registry gets updated when the even already expired and it is cleaned up?
Can you try the following: Use INFO logging and do exactly what you did before. But then after the registry file is not updated, add a new file which is part of a prospector and not related to the other files. This should trigger a state update and clean up your states.
Also on shutdown states are cleaned up / rewritten. If you use info and then stop filebeat, do you get the correct registry?