Elasticsearch not starting - failed to notify ClusterStateListener unable to parse step for policy

Hi,
I run Elasticsearch on liunx (CetnOS7) server in intranet environment.
In the beginning I run Elasticsearch as a cluster with 5 nodes.
I also install Kibana, Logstash, Beats and it run perfectly.

When I try to run as discovery type : single node, it shows
cannot start with [discovery.type] set to [single-node] when local node does not have quorum in voting configuration.

I found this https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416

After I delete all .st file at data/nodes/0/_state and run Elasticsearch again it shows
[2019-08-02][WARN ][o.e.c.s.ClusterApplierService] [hostname] failed to notify ClusterStateListener java.lang.IllegalStateException: unable to parse steps for policy [auditbeat-7.2.0] as it doesn't exist at org.elasticsearch.xpack.indexlifecycle.PolicyStepsRegistry.parseStepsFromPhase(PolicyStepsRegistry.java:146) ~[?:?] at org.elasticsearch.xpack.indexlifecycle.PolicyStepsRegistry.getStep(PolicyStepsRegistry.java:203) ~[?:?] at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.getCurrentStep(IndexLifecycleRunner.java:270) ~[?:?] at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPolicyAfterStateChange(IndexLifecycleRunner.java:204) ~[?:?] at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:268) ~[?:?] at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.clusterChanged(IndexLifecycleService.java:188) ~[?:?] at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$6(ClusterApplierService.java:503) [elasticsearch-7.2.0.jar:7.2.0] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?] at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:500) [elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:477) [elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:418) [elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) [elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.2.0.jar:7.2.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:835) [?:?]

The message in unable to parse steps for policy [XXXX-7.2.0]will change to logstash, all Beats features in different try.

How could I fix it?
Thank you!

It sounds like you tried to reduce a 5-node cluster down to a 1-node cluster but did not follow the instructions for removing master-eligible nodes.

Then, for some strange reason, you deleted a random selection of files from inside the data path. This node is now completely broken. It probably won't recover even if you put the deleted files back again. Your best bet will be to restore it from a snapshot, or just to start again from an empty node.

Hi David,
Thanks for your advice.
But I am not sure what empty node means and how to start again from an empty node.
Could you explain it more detailed?
Thank you!

It means a node with an empty data path. Shut down the node if it's still running, delete all the files in its data path and then start the node up again.

Hi David,
Thank you! It works :slightly_smiling_face:

1 Like