Maybe I missed something: FIlebeat prospectors seems to randomly stop harvesting 'some' files

which filebeat version are you using.

problem is the ignore_older option. We set it to 24h by default. Don't know all in details, but this discussion thread (from message onward) seems to be related.