Yeah, looks like I get what it's doing now.
I've been watching the logs, and looks like logstash is cycling and discovering each log file approximately every second - I think it's because the discovery interval by default is 1 second.
And then it is ignoring some old files cos I configured it to.
But because there are a lot of files (500,000) - it's gonna take probably days 
I keep seeing blocks like the below appearing one after another every second for each discovered file:
[2018-05-04T12:51:19,769][DEBUG][filewatch.discoverer ] Discoverer found file, path: /some/file.log
[2018-05-04T12:51:19,769][DEBUG][filewatch.discoverer ] Discoverer discover_files: /foo/bar*.log: new: /some/file.log (exclude is [])
[2018-05-04T12:51:19,769][DEBUG][filewatch.discoverer ] Discoverer discover_files: /some/file.log: skipping because it was last modified more than 1296000 seconds ago
[2018-05-04T12:51:19,793][DEBUG][filewatch.sincedbcollection] associate: finding: /some/file.log
[2018-05-04T12:51:19,793][DEBUG][filewatch.sincedbcollection] find for path: /some/file.log, found: 'false'
I could double check but I'm pretty sure in the old file input version things got discovered quicker (with the same logstash.conf - I didn't change any config).