Maybe I figured this out. I think you have to run this from PowerShell not the command line. At least on my Windows 2016 Server, it wouldn't work until I used PowerShell to launch it.
My guess would be that the optimize is creating paths that are too deep for the standard command prompt to handle.