I have a single node (without a cluster of several machines) that had an uncontrolled reboot due to power failure. How can I fix this problem?
kibana[4428]: no_shard_available_action_exception: null'. Retrying attempt 2 in 4 seconds.
kibana[4428]: [2023-05-24T13:35:59.009+02:00][INFO ][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> CALCULATE_EXCLUDE_FILTERS. took: 2058ms.
kibana[4428]: [2023-05-24T13:36:03.048+02:00][ERROR][savedobjects-service] [.kibana] Action failed with 'search_phase_execution_exception
kibana[4428]: Root causes:
kibana[4428]: no_shard_available_action_exception: null'. Retrying attempt 3 in 8 seconds.
kibana[4428]: [2023-05-24T13:36:03.049+02:00][INFO ][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> CALCULATE_EXCLUDE_FILTERS. took: 4040ms.
kibana[4428]: [2023-05-24T13:36:11.091+02:00][ERROR][savedobjects-service] [.kibana] Action failed with 'search_phase_execution_exception
kibana[4428]: Root causes:
kibana[4428]: no_shard_available_action_exception: null'. Retrying attempt 4 in 16 seconds.
kibana[4428]: [2023-05-24T13:36:11.092+02:00][INFO ][savedobjects-service] [.kibana] CALCULATE_EXCLUDE_FILTERS -> CALCULATE_EXCLUDE_FILTERS. took: 8043ms.
.kibana_task_manager_8.5.3_001 0 p UNASSIGNED ALLOCATION_FAILED
{
".kibana_task_manager_8.5.3_001" : {
"settings" : {
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "1500"
}
},
"refresh_interval" : "1s",
"hidden" : "true",
"auto_expand_replicas" : "0-1",
"blocks" : {
"write" : "false"
},
"provided_name" : ".kibana_task_manager_8.5.3_001",
"creation_date" : "1672489575889",
"priority" : "10",
"number_of_replicas" : "0",
"uuid" : "XXX",
"version" : {
"created" : "8050399"
},
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
},
"initial_recovery" : {
"_id" : null
}
}
},
"number_of_shards" : "1",
"routing_partition_size" : "1",
"resize" : {
"source" : {
"name" : ".kibana_task_manager_8.5.3_reindex_temp",
"uuid" : "XXX"
}
}
}
}
}
}
I can't remove it:
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [.kibana_task_manager_8.5.3_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [.kibana_task_manager_8.5.3_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}
/_cluster/allocation/explain?pretty
{
"note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
"index" : ".kibana_task_manager_8.5.3_001",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2023-05-24T11:24:38.777Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [JaxNrFlgSFet_CMOtXwhsw]: shard failure, reason [failed to recover from translog], failure [.kibana_task_manager_8.5.3_001/bdWV4XdSRSqSZJD4Q5f0fA][[.kibana_task_manager_8.5.3_001][0]] org.elasticsearch.index.engine.EngineException: failed to recover from translog\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:527)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:500)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:116)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1937)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$8(StoreRecovery.java:471)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:480)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:250)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListener$DelegatingActionListener.onResponse(ActionListener.java:217)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:280)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:289)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1662)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:403)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2288)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog from source [/var/lib/elasticsearch/indices/bdWV4XdSRSqSZJD4Q5f0fA/0/translog/translog-119.tlog] is corrupted, translog truncated\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.readBytes(TranslogSnapshot.java:114)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.BaseTranslogReader.readSize(BaseTranslogReader.java:68)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.readOperation(TranslogSnapshot.java:69)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.next(TranslogSnapshot.java:59)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.MultiSnapshot.next(MultiSnapshot.java:60)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.Translog$SeqNoFilterSnapshot.next(Translog.java:1048)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:1873)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.lambda$openEngineAndRecoverFromTranslog$12(IndexShard.java:1928)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:525)\n\t... 21 more\nCaused by: java.io.EOFException: read past EOF. pos [18327593] length: [4] end: [18327593]\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.io.Channels.readFromFileChannelWithEofException(Channels.java:96)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.readBytes(TranslogSnapshot.java:112)\n\t... 29 more\n",
"last_allocation_status" : "no"
},
"can_allocate" : "no",
"allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster that hold an in-sync copy of its data. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions" : [
{
"node_id" : "JaxNrFlgSFet_CMOtXwhsw",
"node_name" : "XXX",
"transport_address" : "127.0.1.1:9300",
"node_attributes" : {
"ml.allocated_processors" : "32",
"ml.machine_memory" : "67457527808",
"xpack.installed" : "true",
"ml.allocated_processors_double" : "32.0",
"ml.max_jvm_size" : "33285996544"
},
"node_decision" : "no",
"store" : {
"in_sync" : true,
"allocation_id" : "BitfptYDSouZbTqP4aE-ng"
},
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed&metric=none] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2023-05-24T11:24:38.777Z], failed_attempts[5], failed_nodes[[JaxNrFlgSFet_CMOtXwhsw]], delayed=false, last_node[JaxNrFlgSFet_CMOtXwhsw], details[failed shard on node [JaxNrFlgSFet_CMOtXwhsw]: shard failure, reason [failed to recover from translog], failure [.kibana_task_manager_8.5.3_001/bdWV4XdSRSqSZJD4Q5f0fA][[.kibana_task_manager_8.5.3_001][0]] org.elasticsearch.index.engine.EngineException: failed to recover from translog\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:527)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:500)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:116)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1937)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$8(StoreRecovery.java:471)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:480)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:250)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionListener$DelegatingActionListener.onResponse(ActionListener.java:217)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:280)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:289)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1662)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:403)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2288)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog from source [/var/lib/elasticsearch/indices/bdWV4XdSRSqSZJD4Q5f0fA/0/translog/translog-119.tlog] is corrupted, translog truncated\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.readBytes(TranslogSnapshot.java:114)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.BaseTranslogReader.readSize(BaseTranslogReader.java:68)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.readOperation(TranslogSnapshot.java:69)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.next(TranslogSnapshot.java:59)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.MultiSnapshot.next(MultiSnapshot.java:60)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.Translog$SeqNoFilterSnapshot.next(Translog.java:1048)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:1873)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.shard.IndexShard.lambda$openEngineAndRecoverFromTranslog$12(IndexShard.java:1928)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:525)\n\t... 21 more\nCaused by: java.io.EOFException: read past EOF. pos [18327593] length: [4] end: [18327593]\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.common.io.Channels.readFromFileChannelWithEofException(Channels.java:96)\n\tat org.elasticsearch.server@8.7.1/org.elasticsearch.index.translog.TranslogSnapshot.readBytes(TranslogSnapshot.java:112)\n\t... 29 more\n], allocation_status[deciders_no]]]"
}
]
}
]
}
Maybe I need somehow let my cluster know that is alone without other nodes? How do it?