System indices closed, not able to do anything

Elastic Stack Elasticsearch
1

Hi,
by mistake I closed all the indices (even the system indices) of my cluster and now i cannot do anything, not even log into kibana or open them with curl, the credentials (even if correct) are not recognized. how can i re-open the indices?

2

Hi Albe.

Does Elasticsearch properly start for you still?

If so, you can try configuring File-based authentication and using these credentials to run POST /_all/open with curl.

4

Hi Artem,
I tried to follow your suggestion, I added a user with:

usr/share/elasticsearch/bin/elasticsearch-users useradd testelastic -p ******** -r manage,all

but i get this warning:

Warning: The following roles [all,manage] are not in the [/etc/elasticsearch/roles.yml] file. Make sure the names are correct. If the names are correct and t
he roles were created using the API please disregard this message. Nonetheless the user will still be associated with all specified roles
Known roles: [kibana_dashboard_only_user, apm_system, watcher_admin, viewer, logstash_system, rollup_user, kibana_user, beats_admin, remote_monitoring_agent,
rollup_admin, data_frame_transforms_admin, snapshot_user, monitoring_user, enrich_user, kibana_admin, logstash_admin, editor, machine_learning_user, data_fr
ame_transforms_user, machine_learning_admin, watcher_user, apm_user, beats_system, reporting_user, transform_user, kibana_system, transform_admin, transport_
client, remote_monitoring_collector, superuser, ingest_admin]

at the same time when i call the command to open the indices:

curl -X POST "
https://10.145.26.39:9200/_all/_open?pretty" -k -u testelastic:******

I get:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [indices:admin/open] is unauthorized for user [testelastic] with roles [all,manage], this action is granted by the index privilege
s [manage,all]"
      }
    ],
    "type" : "security_exception",
    "reason" : "action [indices:admin/open] is unauthorized for user [testelastic] with roles [all,manage], this action is granted by the index privileges [manage,all]"
  },
  "status" : 403
}

how can i fix this?
Many thanks
Albe

5

Hi Albe,

I see that this user at least has access to the indices, so it's a good start.

What if you assign a superuser role to this user and try to open the indices?

6

I tried but i get this:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "cluster_block_exception",
        "reason" : "index [xxxxxxxxxxxx] blocked by: [FORBIDDEN/6/cluster read-only (api)];index [yyyyyyyyyyyyyyy] blocked by: [FORBIDDEN/6/cluster read-only (ap
i)];index [.monitoring-es-7-2024.05.27] blocked by: [FORBIDDEN/6/cluster read-only (api)];................................."
      }
    ],
        "type" : "cluster_block_exception",
        "reason" : "index [xxxxxxxxxxxx] blocked by: [FORBIDDEN/6/cluster read-only (api)];index [yyyyyyyyyyyyyyy] blocked by: [FORBIDDEN/6/cluster read-only (ap
i)];index [.monitoring-es-7-2024.05.27] blocked by: [FORBIDDEN/6/cluster read-only (api)];................................."
  },
  "status" : 403
}
7

Is your cluster in read-only state too?

If so, you can turn the read-only state off by executing this query:

PUT /_cluster/settings
{
  "persistent" : {
    "cluster.blocks.read_only" : false
  }
}

Also if you have fresh backups of your cluster, rolling back to a fresh backup might be a good last resort option.

8

Hi Artem, it was as you said, it was in read only too. now i was able to fix it.
many thanks

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.