Massive queue in "refresh" thread pool on a single node, causing timeouts

[DavidTurner]

Not to say that there's no locking going on, but we definitely see that the thread is not waiting for a monitor or anything, it's stuck deleting a file for far too long:

@Rayzbam

    0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[PRODUCTION_CRITQUE_2][refresh][T#2]'
     10/10 snapshots sharing following 39 elements
       sun.nio.fs.WindowsNativeDispatcher.DeleteFile0(Native Method)
       sun.nio.fs.WindowsNativeDispatcher.DeleteFile(WindowsNativeDispatcher.java:114)
       sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:249)
       sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
       java.nio.file.Files.delete(Files.java:1126)

sun.nio.fs.WindowsNativeDispatcher.DeleteFile0 is a native method whose implementation in the JDK is this:

openjdk/jdk/blob/jdk-16+36/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c#L261-L268

Pretty much the only thing it could be doing is waiting for DeleteFileW to return.

I don't know a huge amount about debugging this kind of thing in Windows (vs say Linux) but I believe that Process Monitor should show these delete calls taking a long time too.