# Elasticsearch failed Search rejected due to missing shards \[\[.kibana\_task\_manager\_7.17.7\_001\]\[0\]\]
**URL:** https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192
**Category:** Elasticsearch
**Created:** [August 6, 2023, 4:11am UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192 "2023-08-06T04:11:51Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@johnashish](https://discuss.elastic.co/u/johnashish)
#### Post date: [August 6, 2023, 4:11am UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/1 "2023-08-06T04:11:51Z")
Hello,
Current Conf -
Version - Elasticsearch| Kibana - 7.17.3
2 Node Cluster
Recently i am facing lot of trouble to keep the cluster in healthy state.
The error which i am facing is -
```auto
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_task_manager_7.17.7_001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:227) ~[elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) [elasticsearch-7.17.9.jar:7.17.9]
... 267 more
```
```auto
This is my master conf file -
#
cluster.name: monitoring
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ci-sh-mgmt-mon01
node.master: true
node.data: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
path.data: /mon/elasticsearch
#
# Path to log files:
#
#path.logs: /path/to/logs
path.logs: /mon/elasticsearch-logs
path.repo: /mon/snapshots
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.135.0.4
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["0.0.0.0"]
#discovery.seed_hosts: ["10.135.0.4"]
discovery.seed_hosts: ["10.135.0.4"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["ci-sh-mgmt-mon01"]
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
#
# ***WARNING***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don’t have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features.
# Refer to the following documentation for instructions.
#
# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: elastic-stack-ca.p12
```
```auto
This is my Data node conf -
cluster.name: monitoring
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ci-sh-data
node.roles: [data, ingest, remote_cluster_client]
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /mon/elasticsearch-data1
#
# Path to log files:
#
path.logs: /mon/elasticsearch-data1/logs
path.repo: /mon/snapshots
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.135.0.4
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9400
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.135.0.4"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["ci-sh-data"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: elastic-stack-ca.p12
```
This error came for the first time also there are no memory or space issue since i have configured this. There are no restarts of nodes as well.
I have a simple conf nothing complex and using tls certificate for internal node communication.
---
### Author:  [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)
#### Post date: [August 6, 2023, 5:56am UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/2 "2023-08-06T05:56:50Z")
Hi @johnashish
Welcome to the community.
This could happen due to various reasons, such as index corruption or data loss.
To address this issue, you can try the following steps
- Check the status of the affected index using the Elasticsearch cluster health API. Ensure that all the shards for the index are in the "started" state.
- Verify if the index exists and has data. If the index is not present or empty, you may need to reindex the data or restore it from a backup.
- Check the Elasticsearch logs for any errors or warnings related to the index [.kibana\_task\_manager\_7.17.7\_001][0]. This might give you more insights into the root cause of the missing shards.
---
### Author:  [@johnashish](https://discuss.elastic.co/u/johnashish)
#### Post date: [August 6, 2023, 8:14am UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/3 "2023-08-06T08:14:07Z")
Hello @DineshNaik
Thanks for the checking out.
- Only two index are in "Unassigned state", rest all are in Started state.
```auto
{
"cluster_name" : "monitoring",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 374,
"active_shards" : 748,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 99.73333333333333
}
```
Here are the logs -
```auto
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_task_manager_7.17.7_001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:227) ~[elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) [elasticsearch-7.17.9.jar:7.17.9]
... 267 more
[2023-08-06T13:36:54,155][WARN][r.suppressed] [ci-sh-mgmt-mon01] path: /.kibana_task_manager/_update_by_query, params: {ignore_unavailable=true, refresh=true, conflicts=proceed, index=.kibana_task_manager}
org.elasticsearch.action.search.SearchPhaseExecutionException:
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:713) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:199) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:186) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.9.jar:7.17.9]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) [x-pack-security-7.17.9.jar:7.17.9]
at org.elasticsearch.action.ActionListen
```
---
### Author:  [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)
#### Post date: [August 6, 2023, 8:40am UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/4 "2023-08-06T08:40:22Z")
Take a look at this
> [@Shards unassigned for .kibana\_task\_manager index in cluster](https://discuss.elastic.co/t/shards-unassigned-for-kibana-task-manager-index-in-cluster/254147):
>
> Hi, running 3 nodes cluster in prod workload, having problem with .kibana\_task\_manager index in red state, ran cluster/allocation api came to know 2 of the shards unassigned here is the response. for workaround this can be delete and its get recreated when kibana restarted but looking for permanent solution which does not reoccur in future. am trying reproduce this error in lower but we are unable do this. Can someone help me in this regard. { "index":".kibana\_task\_manager", "shard":0, "p…
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [August 6, 2023, 9:03am UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/5 "2023-08-06T09:03:37Z")
It'd be better to link directly to the [section of the manual about troubleshooting missing shards](https://www.elastic.co/guide/en/elasticsearch/reference/current/red-yellow-cluster-status.html) since this contains the most complete and up-to-date information about this problem and what to do about it.
---
### Author:  [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)
#### Post date: [August 9, 2023, 6:05am UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/6 "2023-08-09T06:05:49Z")
@johnashish Did you check the link shared by @DavidTurner ?
---
### Author:  [@johnashish](https://discuss.elastic.co/u/johnashish)
#### Post date: [August 9, 2023, 12:09pm UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/7 "2023-08-09T12:09:06Z")
Yes @DineshNaik i did checked but now i am getting this error in kibana -
I had a backup of indexes which was not working, so manually i put the backup indices inside the data folder - (I have yet to configure snapshots and backup )
But now i am getting error in kibana -
```auto
{"type":"log","@timestamp":"2023-08-09T17:38:04+05:30","tags":["error","plugins","security","session"],"pid":3490,"message":"Failed to schedule session index cleanup task: Saved object index alias [.kibana_task_manager_7.17.7] not found: index_not_found_exception: [index_not_found_exception] Reason: no such index [.kibana_task_manager_7.17.7] and [require_alias] request flag is [true] and [.kibana_task_manager_7.17.7] is not an alias"}
{"type":"log","@timestamp":"2023-08-09T17:38:14+05:30","tags":["error","plugins","security","session"],"pid":3490,"message":"Failed to schedule session index cleanup task: Saved object index alias [.kibana_task_manager_7.17.7] not found: index_not_found_exception: [index_not_found_exception] Reason: no such index [.kibana_task_manager_7.17.7] and [require_alias] request flag is [true] and [.kibana_task_manager_7.17.7] is not an alias"}
```
Is there any way to create this this manually ?
Currently my ES Cluster is in green state
---
### Author:  [@johnashish](https://discuss.elastic.co/u/johnashish)
#### Post date: [August 9, 2023, 1:42pm UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/8 "2023-08-09T13:42:03Z")
Quick Update -
After Kibana restarts, post restart functionality is working fine.
will monitor for next couple of days
Also enabled snapshot and Restore.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 6, 2023, 1:42pm UTC](https://discuss.elastic.co/t/elasticsearch-failed-search-rejected-due-to-missing-shards-kibana-task-manager-7-17-7-001-0/340192/9 "2023-09-06T13:42:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.