Problem with cluster after Windows Server patch

Elastic Search
1

Hi!

A week a go we added a Microsoft patch to our three test servers and after that we have major issues with Elastic. We often get timeout when doing a full reindex.

This is the startup log:

[2025-03-20 13:17:19] [debug] ( prunsrv.c:1994) [ 3020] Apache Commons Daemon procrun log initialized.
[2025-03-20 13:17:19] [info]  ( prunsrv.c:2002) [ 3020] Apache Commons Daemon procrun (1.3.1.0 64-bit) started.
[2025-03-20 13:17:19] [info]  ( prunsrv.c:1906) [ 3020] Running Service 'elasticsearch-service-x64'...
[2025-03-20 13:17:20] [debug] ( prunsrv.c:1677) [ 3676] Inside serviceMain()...
[2025-03-20 13:17:20] [debug] ( prunsrv.c:1123) [ 3676] reportServiceStatusE: dwCurrentState = 2 (SERVICE_START_PENDING), dwWin32ExitCode = 0, dwWaitHint = 3000 milliseconds, dwServiceSpecificExitCode = 0.
[2025-03-20 13:17:20] [info]  ( prunsrv.c:1422) [ 3676] Starting service...
[2025-03-20 13:17:20] [debug] ( prunsrv.c:497 ) [ 3676] Checking Java options for environment variable requirements
[2025-03-20 13:17:20] [debug] ( prunsrv.c:501 ) [ 3676] Checking environment variable requirements for '-XX:+UseSerialGC'
[2025-03-20 13:17:20] [debug] ( prunsrv.c:501 ) [ 3676] Checking environment variable requirements for '-Des.path.home=C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0'
[2025-03-20 13:17:20] [debug] ( prunsrv.c:501 ) [ 3676] Checking environment variable requirements for '-Des.path.conf=C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0\config'
[2025-03-20 13:17:20] [debug] ( prunsrv.c:501 ) [ 3676] Checking environment variable requirements for '-Des.distribution.type=zip'
[2025-03-20 13:17:20] [debug] ( prunsrv.c:501 ) [ 3676] Checking environment variable requirements for '-Dcli.name=windows-service-daemon'
[2025-03-20 13:17:20] [debug] ( prunsrv.c:501 ) [ 3676] Checking environment variable requirements for '-Dcli.libs=lib/tools/server-cli,lib/tools/windows-service-cli'
[2025-03-20 13:17:20] [debug] ( javajni.c:216 ) [ 3676] Explicit RuntimeLib specified 'C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0\jdk\bin\server\jvm.dll'
[2025-03-20 13:17:20] [debug] ( javajni.c:280 ) [ 3676] Adding Java bin path to the PATH to fix loading of awt.dll: 'C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0\jdk\bin'
[2025-03-20 13:17:20] [debug] ( javajni.c:286 ) [ 3676] Loading JVM DLL 'C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0\jdk\bin\server\jvm.dll'
[2025-03-20 13:17:20] [debug] ( javajni.c:414 ) [ 3676] Loaded JVM DLL 'C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0\jdk\bin\server\jvm.dll', home '(null)'.
[2025-03-20 13:17:20] [debug] ( javajni.c:419 ) [ 3676] JNI_GetCreatedJavaVMs...
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[0] -XX:+UseSerialGC
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[1] -Des.path.home=C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[2] -Des.path.conf=C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0\config
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[3] -Des.distribution.type=zip
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[4] -Dcli.name=windows-service-daemon
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[5] -Dcli.libs=lib/tools/server-cli,lib/tools/windows-service-cli
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[6] -Djava.class.path=C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/ecs-logging-core-1.2.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-cli-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-core-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-entitlement-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-geo-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-grok-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-log4j-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-logging-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-lz4-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-native-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-plugin-analysis-api-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-plugin-api-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-secure-sm-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-simdvec-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-tdigest-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/elasticsearch-x-content-8.17.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/HdrHistogram-2.1.9.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/hppc-0.8.1.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/jcodings-1.0.44.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/joni-2.1.29.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/jopt-simple-5.0.2.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/log4j-api-2.19.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/log4j2-ecs-layout-1.2.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-analysis-common-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-backward-codecs-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-core-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-facet-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-grouping-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-highlighter-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-join-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-memory-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-misc-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-queries-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-queryparser-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-sandbox-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lucene-suggest-9.12.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/lz4-java-1.8.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0/lib/cli-launcher/cli-launcher-8.17.0.jar
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[7] exit
[2025-03-20 13:17:20] [debug] ( javajni.c:817 ) [ 4104] JVM Option[8] abort
[2025-03-20 13:17:20] [debug] ( javajni.c:1062) [ 4104] Java worker thread started for org/elasticsearch/launcher/CliToolLauncher:main
[2025-03-20 13:17:20] [debug] ( javajni.c:1068) [ 4104] JNI calling static void method org/elasticsearch/launcher/CliToolLauncher:main
[2025-03-20 13:17:22] [debug] ( prunsrv.c:1486) [ 3676] Java started 'org/elasticsearch/launcher/CliToolLauncher'.
[2025-03-20 13:17:22] [info]  ( prunsrv.c:1585) [ 3676] Service started in 2062 milliseconds.
[2025-03-20 13:17:22] [debug] ( prunsrv.c:1123) [ 3676] reportServiceStatusE: dwCurrentState = 4 (SERVICE_RUNNING), dwWin32ExitCode = 0, dwWaitHint = 0 milliseconds, dwServiceSpecificExitCode = 0.
[2025-03-20 13:17:22] [debug] ( prunsrv.c:1832) [ 3676] Waiting for worker to finish...
[2025-03-20 13:17:22] [debug] ( javajni.c:1157) [ 3676] apxJavaWait -> WaitForSingleObject (0x00000000000002A0, -1 milliseconds) (-1=INFINITE)...
[2025-03-20 13:18:21] [debug] ( javajni.c:1087) [ 4104] Java worker thread finished org/elasticsearch/launcher/CliToolLauncher:main with status = 0
[2025-03-20 13:18:21] [debug] ( javajni.c:1160) [ 3676] apxJavaWait <- WaitForSingleObject (0x00000000000002A0, -1 milliseconds) = 0
[2025-03-20 13:18:21] [debug] ( prunsrv.c:1837) [ 3676] Worker finished.
[2025-03-20 13:18:21] [debug] ( prunsrv.c:1864) [ 3676] Waiting for all threads to exit.
[2025-03-20 13:19:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:19:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:19:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:19:35] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:19:37] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:19:38] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:20:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:20:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:21:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:21:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:22:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:22:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:23:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:23:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:24:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:24:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:24:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:25:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:25:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:26:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:26:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:27:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:27:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:28:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:28:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:29:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:29:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:29:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:30:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:30:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:31:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:31:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:32:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:32:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:33:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:33:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:34:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:34:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:34:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:35:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:35:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:36:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:36:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:37:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:37:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:38:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:38:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:39:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:39:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:39:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:40:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:40:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:41:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:41:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:42:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:42:24] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:43:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:43:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:44:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:44:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:44:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:45:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:45:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:46:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:46:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:47:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:47:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:48:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:48:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:49:22] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:49:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:49:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:50:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:50:25] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:51:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:51:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:52:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:52:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:53:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:53:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:54:23] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:54:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:54:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:55:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:55:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:56:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-20 13:56:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.

What is the Service SERVICE_CONTROL_INTERROGATE signalled.? Could that have something to do with our issues?

On the two servers that not master nodes I find this in the log:

[2025-03-17 13:56:35] [info]  ( prunsrv.c:1621) [ 5956] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-03-17 13:57:05] [info]  ( prunsrv.c:1606) [ 5956] Service SERVICE_CONTROL_STOP signalled.
[2025-03-17 13:57:05] [debug] ( prunsrv.c:1123) [ 5956] reportServiceStatusE: dwCurrentState = 3 (SERVICE_STOP_PENDING), dwWin32ExitCode = 0, dwWaitHint = 3000 milliseconds, dwServiceSpecificExitCode = 0.
[2025-03-17 13:57:05] [info]  ( prunsrv.c:1223) [ 7320] Stopping service...
[2025-03-17 13:57:05] [debug] ( javajni.c:414 ) [ 7320] Loaded JVM DLL 'C:\Program Files\Elastic\Elasticsearch\elasticsearch-8.17.0\jdk\bin\server\jvm.dll', home '(null)'.
[2025-03-17 13:57:05] [debug] ( javajni.c:419 ) [ 7320] JNI_GetCreatedJavaVMs...
[2025-03-17 13:57:05] [debug] ( javajni.c:1062) [ 7620] Java worker thread started for org/elasticsearch/launcher/CliToolLauncher:close
[2025-03-17 13:57:05] [debug] ( javajni.c:1068) [ 7620] JNI calling static void method org/elasticsearch/launcher/CliToolLauncher:close
[2025-03-17 13:57:06] [debug] ( prunsrv.c:1273) [ 7320] Waiting for Java JNI stop worker to finish for org/elasticsearch/launcher/CliToolLauncher:close...
[2025-03-17 13:57:06] [debug] ( javajni.c:1157) [ 7320] apxJavaWait -> WaitForSingleObject (0x00000000000002AC, -1 milliseconds) (-1=INFINITE)...
[2025-03-17 13:57:19] [debug] ( javajni.c:1087) [ 7620] Java worker thread finished org/elasticsearch/launcher/CliToolLauncher:close with status = 0
[2025-03-17 13:57:19] [debug] ( prunsrv.c:1123) [ 5464] reportServiceStatusE: dwCurrentState = 3 (SERVICE_STOP_PENDING), dwWin32ExitCode = 0, dwWaitHint = 0 milliseconds, dwServiceSpecificExitCode = 0.
[2025-03-17 13:57:19] [debug] ( javajni.c:1160) [ 7320] apxJavaWait <- WaitForSingleObject (0x00000000000002AC, -1 milliseconds) = 0
[2025-03-17 13:57:19] [debug] ( prunsrv.c:1278) [ 7320] Java JNI stop worker finished.
[2025-03-17 13:57:19] [debug] ( prunsrv.c:1868) [ 5464] JVM destroyed.
[2025-03-17 13:57:19] [debug] ( prunsrv.c:1123) [ 5464] reportServiceStatusE: dwCurrentState = 1 (SERVICE_STOPPED), dwWin32ExitCode = 0, dwWaitHint = 0 milliseconds, dwServiceSpecificExitCode = 0.
[2025-03-17 13:57:19] [debug] ( prunsrv.c:1123) [ 7320] reportServiceStatusE: dwCurrentState = 3 (SERVICE_STOP_PENDING), dwWin32ExitCode = 0, dwWaitHint = 300000 milliseconds, dwServiceSpecificExitCode = 0.
[2025-03-17 13:57:19] [info]  ( prunsrv.c:1908) [ 5956] Run service finished.
[2025-03-17 13:57:19] [error] ( prunsrv.c:1146) [ 7320] Failed to set service status.
[2025-03-17 13:57:19] [error] ( prunsrv.c:1146) [ 7320] The handle is invalid.
[2025-03-17 13:57:19] [debug] ( prunsrv.c:1386) [ 7320] Waiting for worker to die naturally...
[2025-03-17 13:57:19] [debug] ( prunsrv.c:1397) [ 7320] Worker finished gracefully in 0 milliseconds.
[2025-03-17 13:57:19] [info]  ( prunsrv.c:1407) [ 7320] Service stop thread completed.
[2025-03-17 13:57:19] [info]  ( prunsrv.c:2086) [ 5956] Apache Commons Daemon procrun finished.

Elastic is still running on all the server when I browse http://server1:9200/

Any idea what this issues migth be?

Thanks!

/Kristoffer

2

Hi!

Anyone? We still have this issue, any idea?

Thanks!

/Kristoffer

3

Hi!

Here is an update.
The error:


[2025-03-20 13:53:26] [info]  ( prunsrv.c:1621) [ 3020] Service SERVICE_CONTROL_INTERROGATE signalled.

Only occurs if I run Elastic as a Windows Service. If I start it from the command prompt the error does not show in the log.

I have also upgraded to Elasticsearch 9.0.

Any thougths?

/Kristoffer

4

Have you tried to remove the ES service, and install again?
Stop the elasticsearch process. Run the commands:

bin\elasticsearch-service.bat remove ElasticSearch
bin\elasticsearch-service.bat install ElasticSearch
bin\elasticsearch-service.bat manager ElasticSearch

Check in services what is the correct name for Elasticsearch.

5

Yes, I've done exactly that but still the same error, and it only occurs when Elastic is run as a Windows Service. If started from the command prompt, the error does not occur.

6 
[2025-04-28 08:52:07] [debug] ( javajni.c:419 ) [ 6384] JNI_GetCreatedJavaVMs...
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[0] -XX:+UseSerialGC
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[1] -Des.path.home=C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[2] -Des.path.conf=C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0\config
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[3] -Des.distribution.type=zip
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[4] -Dcli.name=windows-service-daemon
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[5] -Dcli.libs=lib/tools/server-cli,lib/tools/windows-service-cli
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[6] -Djava.class.path=C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/ecs-logging-core-1.2.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-cli-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-core-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-entitlement-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-geo-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-grok-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-log4j-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-logging-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-lz4-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-native-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-plugin-analysis-api-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-plugin-api-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-secure-sm-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-simdvec-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-tdigest-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/elasticsearch-x-content-9.0.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/HdrHistogram-2.1.9.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/hppc-0.8.1.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/jcodings-1.0.44.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/joni-2.1.29.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/jopt-simple-5.0.2.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/log4j-api-2.19.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/log4j2-ecs-layout-1.2.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-analysis-common-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-backward-codecs-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-core-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-facet-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-grouping-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-highlighter-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-join-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-memory-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-misc-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-queries-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-queryparser-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-sandbox-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lucene-suggest-10.1.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/lz4-java-1.8.0.jar;C:\Program Files\Elastic\Elasticsearch\elasticsearch-9.0.0/lib/cli-launcher/cli-launcher-9.0.0.jar
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[7] exit
[2025-04-28 08:52:07] [debug] ( javajni.c:817 ) [ 7972] JVM Option[8] abort
[2025-04-28 08:52:07] [debug] ( javajni.c:1062) [ 7972] Java worker thread started for org/elasticsearch/launcher/CliToolLauncher:main
[2025-04-28 08:52:07] [debug] ( javajni.c:1068) [ 7972] JNI calling static void method org/elasticsearch/launcher/CliToolLauncher:main
[2025-04-28 08:52:08] [debug] ( prunsrv.c:1486) [ 6384] Java started 'org/elasticsearch/launcher/CliToolLauncher'.
[2025-04-28 08:52:08] [info]  ( prunsrv.c:1585) [ 6384] Service started in 1383 milliseconds.
[2025-04-28 08:52:08] [debug] ( prunsrv.c:1123) [ 6384] reportServiceStatusE: dwCurrentState = 4 (SERVICE_RUNNING), dwWin32ExitCode = 0, dwWaitHint = 0 milliseconds, dwServiceSpecificExitCode = 0.
[2025-04-28 08:52:08] [debug] ( prunsrv.c:1832) [ 6384] Waiting for worker to finish...
[2025-04-28 08:52:08] [debug] ( javajni.c:1157) [ 6384] apxJavaWait -> WaitForSingleObject (0x00000000000002A8, -1 milliseconds) (-1=INFINITE)...
[2025-04-28 08:52:34] [debug] ( javajni.c:1087) [ 7972] Java worker thread finished org/elasticsearch/launcher/CliToolLauncher:main with status = 0
[2025-04-28 08:52:34] [debug] ( javajni.c:1160) [ 6384] apxJavaWait <- WaitForSingleObject (0x00000000000002A8, -1 milliseconds) = 0
[2025-04-28 08:52:34] [debug] ( prunsrv.c:1837) [ 6384] Worker finished.
[2025-04-28 08:52:34] [debug] ( prunsrv.c:1864) [ 6384] Waiting for all threads to exit.
[2025-04-28 08:52:42] [info]  ( prunsrv.c:1621) [ 1008] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-04-28 08:52:42] [info]  ( prunsrv.c:1621) [ 1008] Service SERVICE_CONTROL_INTERROGATE signalled.
[2025-04-28 08:53:42] [info]  ( prunsrv.c:1621) [ 1008] Service SERVICE_CONTROL_INTERROGATE signalled.
7

The service has been started. Others are info messages. Can you check does the process run on TCP 9200? Use netstat or curl to check

8

Yes, the service is up and working just fine so maybe that info message is nothing to worry about?

9

If is working in the services mode, don't see any need to investigate. Maybe someone from Elastic team has more ideas about this.

10

That's good to hear, but how does this square with earlier observation

Are those issue also resolved?

11

Hi!

No, we still get the timeouts and I thought that the service info message might have something to do with it, perhaps that is not the case?

/Kristoffer

12

Timeouts in the elastic log or from Kibana or app?

13

Timeout in the .net core application that calls Elastic

14

You can follow next steps:

  1. Run ES as service to be a stable. There should be a trace something like this
    [2025-04-28T10:34:23,928][INFO ][o.e.h.AbstractHttpServerTransport] [elk1] publish_address {192.168.1.111:9200}, bound_addresses {[::]:9200}
    [2025-04-28T10:34:27,280][INFO ][o.e.c.r.a.AllocationService] [elk1] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.ds-.logs-elasticsearch.deprecation-default-2025.04.17-000001][0]]])." previous.health="RED" reason="shards started [[.ds-.logs-elasticsearch.deprecation-default-2025.04.17-000001][0]]"

  2. Run elasticsearch-service.bat manager Elasticsearch. Review "Java virtual Machine" and "Java Classpath",check are path correct.
    There was a bug with the drive's capital letter on Windows, but I don't think is your case. Check and exclude a possibility.
    Check how the service does start, default is the account Local System account

  3. Use curl to test ES APIs. Run it on every node, maybe one of them or index has an issue.
    GET 'http://server1:9200/_cat/health
    GET http://server1:9200/_cat/indices?bytes=b&s=store.size:desc&v&pretty

  4. Might be issues with the local firewall, check opened ports.

  5. Set log level to info, not debug. It's flood the log and slow down.

  6. Check do you have enough disk space. Reindexing is like index cloning + little bit additional space. How large are indices?

  7. Can you copy more lines, instead of 1-2? The best option is from the start until a problem show up.You can put on gdrive or A similar location.

15

Thanks for alla the tips, I will try them.

I can now confirm that the info message has to do with the server patch.
This nigth our productions servers, who did not have the messages, was updated with the latest Windows Server patch and after restart the message started to appear in the logs.
Not sure the message is a problem, just curious what is means?

/Kristoffer

16

If you started servers in the debug mode, that might mean someone had changed the log level and after restart, changes were applied. Not sure how the node had been restarted, removing/adding the cluster with stopping services or simply restarted.
Also might be on ES side, however ES use Java not .Net or simply something has been changed long time before restart, and after restart changes were applied. Might be Java, the env. settings, etc.

In any case, you can open a new issue with much more information than here.

17

Ok, I'll look into to open an issue.

These are the Windows patches:
KB5055519 | April 8, 2025—KB5055519 (OS Build 17763.7136) - Microsoft Support

KB5055175 | April 8, 2025-KB5055175 Cumulative Update for .NET Framework 3.5 and 4.8 for Windows 10, version 1809 and Windows Server 2019 - Microsoft Support

/Kristoffer