Index routing and deletion

[parthmaniar (Parth D. Maniar)]

Hello,

I hope my message finds community members and their loved ones safe and healthy.

I have a three-node cluster with two nodes storing data and one being a voting-only node.

The current status of the cluster is yellow because two indices are not assigned.

index                                                                     shard prirep state      node          unassigned.reason
.ds-logs-endpoint.events.process-default-2023.07.06-000088                0     r      UNASSIGNED               INDEX_CREATED
.kibana-event-log-8.8.2-000001                                            0     r      UNASSIGNED               INDEX_CREATED

I have enabled routing after the recent 8.8.1 update using.

PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.enable" : null
  }
}

However, the cluster is still yellow. Hence my first question How do I have these indices routed and replicas created?

My second question is that I have a total of 1100 primary shards. However, ~800 (~73%) are indices with . at the start of the name.

{
    "cluster_name": "data_analytics_1",
    "status": "yellow",
    "timed_out": false,
    "number_of_nodes": 3,
    "number_of_data_nodes": 2,
    "active_primary_shards": 1100,
    "active_shards": 2198,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 2,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 99.90909090909092
}

Is it safe to delete these? Especially of older versions such as .ds-logs-elastic_agent.filebeat-default-2021.07.16-000007 or .ds-.logs-deprecation.elasticsearch-default-2022.10.14-000044 which are clearly of older versions?

[parthmaniar (Parth D. Maniar)]

Part of this question is answered.

I used the wrong API to get the status. The correct query for this problem is:

GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*
{
  "index": "{index name}",
  "shard": 0,
  "primary": false
}

The problem in my case was that one of the nodes was running 8.8.1 (secondarynode) while the other (primary) was running 8.8.2. I am using Ubuntu as the OS and after a reboot it was solved (there was no pending package update).

Thank you.