Cluster with all system indices in RED status

Elastic Stack Elasticsearch
1

Hello,

I have a cluster where i lost some nodes that are storing primaries & replicas of all system indices.

index shard prirep state
.reporting-2022-09-18 0 p UNASSIGNED
.reporting-2022-09-18 0 r UNASSIGNED
.kibana-event-log-8.3.3-000002 0 p UNASSIGNED
.kibana-event-log-8.3.3-000002 0 r UNASSIGNED
.ds-.slm-history-5-2022.11.06-000003 0 p UNASSIGNED
.ds-.slm-history-5-2022.11.06-000003 0 r UNASSIGNED
.ds-ilm-history-5-2022.09.02-000001 0 p UNASSIGNED
.ds-ilm-history-5-2022.09.02-000001 0 r UNASSIGNED
.async-search 0 p UNASSIGNED
.async-search 0 r UNASSIGNED
.kibana-event-log-8.3.3-000001 0 p UNASSIGNED
.kibana-event-log-8.3.3-000001 0 r UNASSIGNED
.ds-.logs-deprecation.elasticsearch-default-2022.10.02-000002 0 p UNASSIGNED
.ds-.logs-deprecation.elasticsearch-default-2022.10.02-000002 0 r UNASSIGNED
.tasks 0 p UNASSIGNED
.tasks 0 r UNASSIGNED
.kibana-event-log-8.3.3-000003 0 p UNASSIGNED
.kibana-event-log-8.3.3-000003 0 r UNASSIGNED
.security-7 0 p UNASSIGNED
.security-7 0 r UNASSIGNED
.apm-agent-configuration 0 p UNASSIGNED
.apm-agent-configuration 0 r UNASSIGNED
.kibana_8.3.3_001 0 p UNASSIGNED
.kibana_8.3.3_001 0 r UNASSIGNED
.transform-internal-007 0 p UNASSIGNED
.transform-internal-007 0 r UNASSIGNED
.ds-ilm-history-5-2022.10.02-000002 0 p UNASSIGNED
.ds-ilm-history-5-2022.10.02-000002 0 r UNASSIGNED
.ds-.logs-deprecation.elasticsearch-default-2022.11.01-000003 0 p UNASSIGNED
.ds-.logs-deprecation.elasticsearch-default-2022.11.01-000003 0 r UNASSIGNED
.ds-.slm-history-5-2022.10.07-000002 0 p UNASSIGNED
.ds-.slm-history-5-2022.10.07-000002 0 r UNASSIGNED
.ds-ilm-history-5-2022.11.01-000003 0 p UNASSIGNED
.ds-ilm-history-5-2022.11.01-000003 0 r UNASSIGNED
.transform-notifications-000002 0 p UNASSIGNED
.transform-notifications-000002 0 r UNASSIGNED
.geoip_databases 0 p UNASSIGNED
.geoip_databases 0 r UNASSIGNED
.kibana_task_manager_8.3.3_001 0 p UNASSIGNED
.kibana_task_manager_8.3.3_001 0 r UNASSIGNED
.apm-custom-link 0 p UNASSIGNED
.apm-custom-link 0 r UNASSIGNED
.ds-.slm-history-5-2022.09.07-000001 0 p UNASSIGNED
.ds-.slm-history-5-2022.09.07-000001 0 r UNASSIGNED
.reporting-2022-11-06 0 p UNASSIGNED
.reporting-2022-11-06 0 r UNASSIGNED
.kibana_security_session_1 0 p UNASSIGNED
.kibana_security_session_1 0 r UNASSIGNED

Is there any way to delete these system indexes and restart cluster/kibana to get them created newly

I tried to create an emergency user to be able to access cluster :slight_smile:

bin/elasticsearch-users useradd restore_user -p e****** -r superuser

But not able to delete them

curl -k -u restore_user:*****  -X DELETE "https://*******:9200/.security-7?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [indices:admin/delete] is unauthorized for user [restore_kbn] with roles [kibana_system] on restricted indices [.security-7], this action is granted by the index privileges [delete_index,manage,all]"
      }
    ],
    "type" : "security_exception",
    "reason" : "action [indices:admin/delete] is unauthorized for user [restore_user] with roles [kibana_system] on restricted indices [.security-7], this action is granted by the index privileges [delete_index,manage,all]"
  },
  "status" : 403
}
2

Did this work ok? What was the response from Elasticsearch for it?

3

Yes the useradd worked and output was

{
  "created": true 
}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.