Elastich search:unassigned shards: status yellow

Elastic Stack Kibana
1

Hello Team,

My team is trying to create an Elasticsearch -cluster with Kibana.
on one server Elasticsearch node works fine with zero shards.
But another server is complaining about the status "yellow".
Below is the unassigned shards ID. Can you please guide us on How to remove/assign/recreate the correct assigned shards?

metricbeat-7.17.10-2023.07.11-000001 0 r UNASSIGNED CLUSTER_RECOVERED

Regards,
Deepika

2

This is a replica shard, which can not be allocated on the same Elasticsearch node as the primary shard. If the node with the primary shard goes away, the data will be lost. To make sure that doesn't happen, you could deploy a 2nd instance of Elasticsearch in the cluster. The replica shard will be assigned automatically to the new node. When that happens, the cluster state should be green.

3

Hello Tim,

Thanks for your quick reply and for considering my request!

We have three nodes, one is dedicated to Kibana, and two other nodes for ELK, one is master and the other is a slave.
the first node is working fine with the status "Green"
The second node is complaining about the status "Yellow"
and we have to make it work with these three nodes only for now.
so we cant create a new node for elk.
how we can do it with this only?

Regards,
Deepika

4

I think I understand now. You have two Elasticsearch nodes that are not joined together in a cluster. One node has no shards and it's cluster state is green. Another node needs to assign shards so it's cluster state is yellow.

See: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/add-elasticsearch-nodes.html

5

Thanks once again for your prompt reply.

somehow we made both nodes green(the yellow node is showing green because we stopped metric beats services).

Elk stack version: 7.17

First Scenario:
when we are assigning cluster nodes of Elasticsearch in Kibana: Kibana is throwing the error "504 Gateway time out"

Second scenario:
With Individual node: Kibana works fine* (**Page is loading on the browser with Elasticsearch).

Could you please guide us why with two nodes, getting "504 Gateway time out?

Please refer error below for more reference.

"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["info","savedobjects-service"],"pid":21832,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["info","savedobjects-service"],"pid":21832,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 6ms."}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["info","savedobjects-service"],"pid":21832,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 15ms."}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["info","savedobjects-service"],"pid":21832,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 4ms."}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["info","savedobjects-service"],"pid":21832,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 13ms."}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["error","savedobjects-service"],"pid":21832,"message":"[.kibana_task_manager] Unexpected Elasticsearch ResponseError: statusCode: 404, method: POST, url: /_search?allow_partial_search_results=false error: [index_not_found_exception]: no such index [.kibana_task_manager_7.17.10_001],"}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["fatal","root"],"pid":21832,"message":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 404, method: POST, url: /_search?allow_partial_search_results=false error: [index_not_found_exception]: no such index [.kibana_task_manager_7.17.10_001],\n at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:164:13)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Promise.all (index 1)\n at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n at Server.start (/usr/share/kibana/src/core/server/server.js:330:31)\n at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)\n at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)\n at Command. (/usr/share/kibana/src/cli/serve/serve.js:229:5)"}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["info","plugins-system","standard"],"pid":21832,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":21832,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2023-07-14T10:08:53+02:00","tags":["error","savedobjects-service"],"pid":21832,"message":"[.kibana] Unexpected Elasticsearch ResponseError: statusCode: 404, method: POST, url: /_search?allow_partial_search_results=false error: [index_not_found_exception]: no such index [.kibana_7.17.10_001],"}

Regards,
Deepika

6

Thanks for your help/support.

ELk started working now after deleting logs from /var/lib/elasticsearch(all index files)

Regard,
Deepika

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.