I would expect your config above to do exactly what you suggest above, just that max could probably be 2s. Perhaps check your logs do see what exactly happens when and where the delay is.
It is possible to even set the above values lower, but you will get a greater CPU overhead as it constantly checks the files.