Elasticsearch 7.16.1 crashing randomly

Elastic Stack Elasticsearch
21

@Keith_Massey I found some logs:

This is another cluster with 3 master(-only) nodes and some data-nodes.

When I restarted the master-node, the two other master-eligible-nodes crashed with identical logs sequentially.

fatal error in thread [elasticsearch[logindexer-3][clusterApplierService#updateTask][T#1]], exiting
java.lang.NoClassDefFoundError: org/elasticsearch/common/xcontent/XContent
	at org.elasticsearch.xpack.ilm.PolicyStepsRegistry.update(PolicyStepsRegistry.java:133)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.applyClusterState(IndexLifecycleService.java:323)
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:553)
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:540)
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:503)
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:428)
	at org.elasticsearch.cluster.service.ClusterApplierService.access$000(ClusterApplierService.java:56)
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:154)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.xcontent.XContent
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:904)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 14 more
ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elkng-gce-int.log

Is this helpful?