Massive queue in "refresh" thread pool on a single node, causing timeouts

DavidTurner · August 11, 2021, 6:53pm

Not to say that there's no locking going on, but we definitely see that the thread is not waiting for a monitor or anything, it's stuck deleting a file for far too long:

Rayzbam:

    0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[PRODUCTION_CRITQUE_2][refresh][T#2]'
     10/10 snapshots sharing following 39 elements
       sun.nio.fs.WindowsNativeDispatcher.DeleteFile0(Native Method)
       sun.nio.fs.WindowsNativeDispatcher.DeleteFile(WindowsNativeDispatcher.java:114)
       sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:249)
       sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
       java.nio.file.Files.delete(Files.java:1126)

sun.nio.fs.WindowsNativeDispatcher.DeleteFile0 is a native method whose implementation in the JDK is this:

github.com

openjdk/jdk/blob/jdk-16+36/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c#L261-L268


      
          JNIEXPORT void JNICALL
          Java_sun_nio_fs_WindowsNativeDispatcher_DeleteFile0(JNIEnv* env, jclass this, jlong address)
          {
              LPCWSTR lpFileName = jlong_to_ptr(address);
              if (DeleteFileW(lpFileName) == 0) {
                  throwWindowsException(env, GetLastError());
              }
          }

Pretty much the only thing it could be doing is waiting for DeleteFileW to return.

I don't know a huge amount about debugging this kind of thing in Windows (vs say Linux) but I believe that Process Monitor should show these delete calls taking a long time too.