Hi
I have a test server that is operated as a single cluster node. However, I don't want to lose the information I have collected.
I'm just not experienced enough with Elastic to figure out the root cause 
Health
GET _cluster/health?filter_path=status,*_shards
{
"status": "red",
"active_primary_shards": 117,
"active_shards": 117,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 57,
"delayed_unassigned_shards": 0
}
All UNASSIGNED States
_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state
index shard prirep state node unassigned.reason
.ds-.monitoring-es-8-mb-2023.09.05-000007 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.10.08-000018 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.05.05-000002 0 r UNASSIGNED CLUSTER_RECOVERED
.kibana_task_manager_8.7.0_001 0 p UNASSIGNED ALLOCATION_FAILED
.ds-.monitoring-es-8-mb-2023.09.23-000013 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2024.02.06-000005 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.07.04-000004 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2024.04.06-000007 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.10.02-000007 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.08-000008 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.17-000011 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.05.05-000002 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.07.04-000004 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2023.11.08-000002 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.02-000006 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.06.04-000003 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2024.01.07-000004 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2023.10.09-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2023.10.09-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2024.02.06-000005 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2023.12.08-000003 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.14-000010 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.10.02-000016 0 r UNASSIGNED CLUSTER_RECOVERED
.lists-default-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.10.05-000017 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.08.03-000005 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2024.05.06-000008 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2024.03.07-000006 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.09.02-000006 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2024.07.05-000010 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.06.04-000003 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.08.30-000005 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.11.01-000008 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.09.02-000006 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.29-000015 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.08.18-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.11-000009 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.08.24-000003 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2024.06.05-000009 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.08.03-000005 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.04.05-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2024.07.05-000010 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2024.04.06-000007 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2024.01.07-000004 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.08.21-000002 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2024.06.05-000009 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.11.01-000008 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.7.0-2023.10.02-000007 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.08.27-000004 0 r UNASSIGNED CLUSTER_RECOVERED
.items-default-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2023.12.08-000003 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2024.03.07-000006 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-winlogbeat-8.10.2-2024.05.06-000008 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.10.2-2023.11.08-000002 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.20-000012 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-metricbeat-8.7.0-2023.04.05-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.monitoring-es-8-mb-2023.09.26-000014 0 r UNASSIGNED CLUSTER_RECOVERED
The main problem is probably:
.kibana_task_manager_8.7.0_001 0 p UNASSIGNED ALLOCATION_FAILED
Index-Information
.kibana_task_manager_8.7.0_001/_settings?flat_settings=true&include_defaults=true
{
".kibana_task_manager_8.7.0_001": {
"settings": {
"index.auto_expand_replicas": "0-1",
"index.creation_date": "1680707800346",
"index.hidden": "true",
"index.mapping.total_fields.limit": "1500",
"index.number_of_replicas": "0",
"index.number_of_shards": "1",
"index.priority": "10",
"index.provided_name": ".kibana_task_manager_8.7.0_001",
"index.refresh_interval": "1s",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "01e3RjVVSziSHFu8vGieBw",
"index.version.created": "8070099"
},
"defaults": {
"index.allocation.existing_shards_allocator": "gateway_allocator",
"index.allocation.max_retries": "5",
"index.analyze.max_token_count": "10000",
"index.blocks.metadata": "false",
"index.blocks.read": "false",
"index.blocks.read_only": "false",
"index.blocks.read_only_allow_delete": "false",
"index.blocks.write": "false",
"index.bloom_filter_for_id_field.enabled": "true",
"index.codec": "default",
"index.compound_format": "1gb",
"index.data_path": "",
"index.default_pipeline": "_none",
"index.downsample.origin.name": "",
"index.downsample.origin.uuid": "",
"index.downsample.source.name": "",
"index.downsample.source.uuid": "",
"index.downsample.status": "unknown",
"index.fast_refresh": "false",
"index.fielddata.cache": "node",
"index.final_pipeline": "_none",
"index.flush_after_merge": "512mb",
"index.force_memory_term_dictionary": "false",
"index.format": "0",
"index.frozen": "false",
"index.gc_deletes": "60s",
"index.highlight.max_analyzed_offset": "1000000",
"index.highlight.weight_matches_mode.enabled": "true",
"index.indexing.slowlog.level": "TRACE",
"index.indexing.slowlog.reformat": "true",
"index.indexing.slowlog.source": "1000",
"index.indexing.slowlog.threshold.index.debug": "-1",
"index.indexing.slowlog.threshold.index.info": "-1",
"index.indexing.slowlog.threshold.index.trace": "-1",
"index.indexing.slowlog.threshold.index.warn": "-1",
"index.lifecycle.indexing_complete": "false",
"index.lifecycle.name": "",
"index.lifecycle.origination_date": "-1",
"index.lifecycle.parse_origination_date": "false",
"index.lifecycle.prefer_ilm": "true",
"index.lifecycle.rollover_alias": "",
"index.lifecycle.step.wait_time_threshold": "12h",
"index.load_fixed_bitset_filters_eagerly": "true",
"index.look_ahead_time": "2h",
"index.mapping.coerce": "false",
"index.mapping.depth.limit": "20",
"index.mapping.dimension_fields.limit": "21",
"index.mapping.field_name_length.limit": "9223372036854775807",
"index.mapping.ignore_malformed": "false",
"index.mapping.nested_fields.limit": "50",
"index.mapping.nested_objects.limit": "10000",
"index.max_adjacency_matrix_filters": "100",
"index.max_docvalue_fields_search": "100",
"index.max_inner_result_window": "100",
"index.max_ngram_diff": "1",
"index.max_refresh_listeners": "1000",
"index.max_regex_length": "1000",
"index.max_rescore_window": "10000",
"index.max_result_window": "10000",
"index.max_script_fields": "32",
"index.max_shingle_diff": "3",
"index.max_slices_per_scroll": "1024",
"index.max_terms_count": "65536",
"index.merge.policy.deletes_pct_allowed": "20.0",
"index.merge.policy.expunge_deletes_allowed": "10.0",
"index.merge.policy.floor_segment": "2mb",
"index.merge.policy.max_merge_at_once": "10",
"index.merge.policy.max_merge_at_once_explicit": "30",
"index.merge.policy.max_merged_segment": "0b",
"index.merge.policy.merge_factor": "32",
"index.merge.policy.segments_per_tier": "10.0",
"index.merge.policy.type": "UNSET",
"index.merge.scheduler.auto_throttle": "true",
"index.merge.scheduler.max_merge_count": "7",
"index.merge.scheduler.max_thread_count": "2",
"index.mode": "standard",
"index.number_of_routing_shards": "1",
"index.optimize_auto_generated_id": "true",
"index.override_write_load_forecast": "0.0",
"index.percolator.map_unmapped_fields_as_text": "false",
"index.queries.cache.enabled": "true",
"index.query.default_field": [
"*"
],
"index.query.parse.allow_unmapped_fields": "true",
"index.query_string.lenient": "false",
"index.recovery.type": "",
"index.requests.cache.enable": "true",
"index.rollup.source.name": "",
"index.rollup.source.uuid": "",
"index.routing.allocation.disk.watermark.ignore": "false",
"index.routing.allocation.enable": "all",
"index.routing.allocation.total_shards_per_node": "-1",
"index.routing.rebalance.enable": "all",
"index.routing_partition_size": "1",
"index.routing_path": [],
"index.search.idle.after": "30s",
"index.search.slowlog.level": "TRACE",
"index.search.slowlog.threshold.fetch.debug": "-1",
"index.search.slowlog.threshold.fetch.info": "-1",
"index.search.slowlog.threshold.fetch.trace": "-1",
"index.search.slowlog.threshold.fetch.warn": "-1",
"index.search.slowlog.threshold.query.debug": "-1",
"index.search.slowlog.threshold.query.info": "-1",
"index.search.slowlog.threshold.query.trace": "-1",
"index.search.slowlog.threshold.query.warn": "-1",
"index.search.throttled": "false",
"index.shard.check_on_startup": "false",
"index.shard_limit.group": "normal",
"index.soft_deletes.enabled": "true",
"index.soft_deletes.retention.operations": "0",
"index.soft_deletes.retention_lease.period": "12h",
"index.sort.field": [],
"index.sort.missing": [],
"index.sort.mode": [],
"index.sort.order": [],
"index.source_only": "false",
"index.store.fs.fs_lock": "native",
"index.store.preload": [],
"index.store.snapshot.blob_cache.metadata_files.max_length": "64kb",
"index.store.snapshot.cache.enabled": "true",
"index.store.snapshot.cache.excluded_file_types": [],
"index.store.snapshot.cache.prewarm.enabled": "true",
"index.store.snapshot.delete_searchable_snapshot": "false",
"index.store.snapshot.index_name": "",
"index.store.snapshot.index_uuid": "",
"index.store.snapshot.partial": "false",
"index.store.snapshot.repository_name": "",
"index.store.snapshot.repository_uuid": "",
"index.store.snapshot.snapshot_name": "",
"index.store.snapshot.snapshot_uuid": "",
"index.store.snapshot.uncached_chunk_size": "-1b",
"index.store.stats_refresh_interval": "10s",
"index.store.type": "",
"index.time_series.end_time": "9999-12-31T23:59:59.999Z",
"index.time_series.es87tsdb_codec.enabled": "true",
"index.time_series.start_time": "-9999-01-01T00:00:00Z",
"index.top_metrics_max_size": "10",
"index.translog.durability": "REQUEST",
"index.translog.flush_threshold_age": "1m",
"index.translog.flush_threshold_size": "10gb",
"index.translog.generation_threshold_size": "64mb",
"index.translog.retention.age": "-1",
"index.translog.retention.size": "-1",
"index.translog.sync_interval": "5s",
"index.unassigned.node_left.delayed_timeout": "1m",
"index.verified_before_close": "false",
"index.version.compatibility": "8070099",
"index.warmer.enabled": "true",
"index.write.wait_for_active_shards": "1",
"index.xpack.ccr.following_index": "false",
"index.xpack.version": "",
"index.xpack.watcher.template.version": ""
}
}
}
_cluster/allocation/explain
{
"index": ".kibana_task_manager_8.7.0_001",
"shard": 0,
"primary": true
}
{
"index": ".kibana_task_manager_8.7.0_001",
"shard": 0,
"primary": true,
"current_state": "unassigned",
"unassigned_info": {
"reason": "ALLOCATION_FAILED",
"at": "2024-07-26T16:08:43.086Z",
"failed_allocation_attempts": 5,
"details": "failed shard on node [6ncxkK-IST-_InDRaL4-0g]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.kibana_task_manager_8.7.0_001][0]: Recovery failed on {WXTASK2P}{6ncxkK-IST-_InDRaL4-0g}{QKfxk5RATsGSfU-PejHsVw}{WXTASK2P}{172.20.249.89}{172.20.249.89:9300}{cdfhilmrstw}{8.10.2}{7000099-8100299}{ml.machine_memory=68718444544, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=10.0.0, ml.max_jvm_size=33285996544, ml.allocated_processors_double=4.0, ml.allocated_processors=4}\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:392)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:485)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.CompositeIndexEventListener.callListeners(CompositeIndexEventListener.java:278)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:287)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:314)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1682)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:410)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2338)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\r\nCaused by: [.kibana_task_manager_8.7.0_001/01e3RjVVSziSHFu8vGieBw][[.kibana_task_manager_8.7.0_001][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway\r\n\t... 16 more\r\nCaused by: [.kibana_task_manager_8.7.0_001/01e3RjVVSziSHFu8vGieBw][[.kibana_task_manager_8.7.0_001][0]] org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:274)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:224)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:14)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2006)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1970)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:479)\r\n\t... 15 more\r\nCaused by: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=959467631 vs expected header=1071082519 (resource=MemorySegmentIndexInput(path=\"C:\\_Mon\\e\\data\\indices\\01e3RjVVSziSHFu8vGieBw\\0\\index\\_8rh96.cfs\"))\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:186)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:80)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:86)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1244)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1262)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1117)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2589)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2577)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:267)\r\n\t... 20 more\r\n",
"last_allocation_status": "no"
},
"can_allocate": "no",
"allocate_explanation": "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster that hold an in-sync copy of its data. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions": [
{
"node_id": "6ncxkK-IST-_InDRaL4-0g",
"node_name": "WXTASK2P",
"transport_address": "172.20.249.89:9300",
"node_attributes": {
"ml.machine_memory": "68718444544",
"transform.config_version": "10.0.0",
"xpack.installed": "true",
"ml.config_version": "10.0.0",
"ml.max_jvm_size": "33285996544",
"ml.allocated_processors_double": "4.0",
"ml.allocated_processors": "4"
},
"node_decision": "no",
"store": {
"in_sync": true,
"allocation_id": "wAP4hMd5RwW6b2o7HsdoKQ"
},
"deciders": [
{
"decider": "max_retry",
"decision": "NO",
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed&metric=none] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2024-07-26T16:08:43.086Z], failed_attempts[5], failed_nodes[[6ncxkK-IST-_InDRaL4-0g]], delayed=false, last_node[6ncxkK-IST-_InDRaL4-0g], details[failed shard on node [6ncxkK-IST-_InDRaL4-0g]: failed recovery, failure org.elasticsearch.indices.recovery.RecoveryFailedException: [.kibana_task_manager_8.7.0_001][0]: Recovery failed on {WXTASK2P}{6ncxkK-IST-_InDRaL4-0g}{QKfxk5RATsGSfU-PejHsVw}{WXTASK2P}{172.20.249.89}{172.20.249.89:9300}{cdfhilmrstw}{8.10.2}{7000099-8100299}{ml.machine_memory=68718444544, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=10.0.0, ml.max_jvm_size=33285996544, ml.allocated_processors_double=4.0, ml.allocated_processors=4}\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$25(IndexShard.java:3201)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$8(StoreRecovery.java:392)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:485)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.CompositeIndexEventListener.callListeners(CompositeIndexEventListener.java:278)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:287)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:314)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1682)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:410)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2338)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\r\nCaused by: [.kibana_task_manager_8.7.0_001/01e3RjVVSziSHFu8vGieBw][[.kibana_task_manager_8.7.0_001][0]] org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway\r\n\t... 16 more\r\nCaused by: [.kibana_task_manager_8.7.0_001/01e3RjVVSziSHFu8vGieBw][[.kibana_task_manager_8.7.0_001][0]] org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:274)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:224)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:14)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2006)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1970)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:479)\r\n\t... 15 more\r\nCaused by: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=959467631 vs expected header=1071082519 (resource=MemorySegmentIndexInput(path=\"C:\\_Mon\\e\\data\\indices\\01e3RjVVSziSHFu8vGieBw\\0\\index\\_8rh96.cfs\"))\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:186)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:253)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:80)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:86)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1244)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1262)\r\n\tat org.apache.lucene.core@9.7.0/org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1117)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2589)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2577)\r\n\tat org.elasticsearch.server@8.10.2/org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:267)\r\n\t... 20 more\r\n], allocation_status[deciders_no]]]"
}
]
}
]
}