# Search phase execution exception in kibana canvas **URL:** \ **Category:** Kibana\ **Created:** [June 20, 2019, 8:05am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622 "2019-06-20T08:05:19Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [June 20, 2019, 8:05am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/1 "2019-06-20T08:05:20Z")
Hi , Using canvas in kibana tried to display metrics for a server but shows an error "[essql] \> Unexpected error from Elasticsearch: search phase execution exception" . Below is the message from kibana.stdout "2019-06-20T08:02:02Z","tags":["error","task\_manager"],"pid":22254,"message":"Failed to poll for work: [cluster\_block\_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana\_task\_manager/\_doc/Maps-maps\_telemetry/\_update","query":{"if\_seq\_no":40,"if\_primary\_term":4,"refresh":"true"},"body":"{\"doc\":{\"type\":\"task\",\"task\":{\"taskType\":\"maps\_telemetry\",\"state\":\"{\\\"runs\\\":1,\\\"stats\\\":{}}\",\"params\":\"{}\",\"attempts\":0,\"scheduledAt\":\"2019-05-27T04:27:32.931Z\",\"runAt\":\"2019-06-20T08:03:02.997Z\",\"status\":\"running\"},\"kibana\":{\"uuid\":\"979cbc12-fc31-443f-9583-0071fb272f4b\",\"version\":6070299,\"apiVersion\":1}}}","statusCode":403,"response":"{\"error\":{\"root\_cause\":[{\"type\":\"cluster\_block\_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster\_block\_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"} {"type":"log","@timestamp":"2019-06-20T08:02:06Z","tags":["error","task\_manager"],"pid":22254,"message":"Failed to poll for work: [cluster\_block\_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana\_task\_manager/\_doc/Maps-maps\_telemetry/\_update","query":{"if\_seq\_no":40,"if\_primary\_term":4,"refresh":"true"},"body":"{\"doc\":{\"type\":\"task\",\"task\":{\"taskType\":\"maps\_telemetry\",\"state\":\"{\\\"runs\\\":1,\\\"stats\\\":{}}\",\"params\":\"{}\",\"attempts\":0,\"scheduledAt\":\"2019-05-27T04:27:32.931Z\",\"runAt\":\"2019-06-20T08:03:06.022Z\",\"status\":\"running\"},\"kibana\":{\"uuid\":\"979cbc12-fc31-443f-9583-0071fb272f4b\",\"version\":6070299,\"apiVersion\":1}}}","statusCode":403,"response":"{\"error\":{\"root\_cause\":[{\"type\":\"cluster\_block\_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster\_block\_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"} ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [June 20, 2019, 11:03am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/2 "2019-06-20T11:03:49Z")
Looks like the user doesn't have enough permissions to do these queries. What user are you using to login and what roles do you have assigned? ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [June 21, 2019, 2:46am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/3 "2019-06-21T02:46:29Z")
> [@Marius\_Dragomir](#): > > What user are you using to login and what roles do you have assigned? Hi Marius, We have not defined any user and roles as i see user, roles.yml and user\_roles are empty ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [June 21, 2019, 9:16am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/4 "2019-06-21T09:16:11Z")
Did you enable security in Kibana or Elasticsearch? ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [June 24, 2019, 5:00am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/5 "2019-06-24T05:00:22Z")
Hi Marius, We have not enabled security in kibana or elasticsearch ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [June 24, 2019, 2:47pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/6 "2019-06-24T14:47:37Z")
Then maybe your ES cluster ran out of space and marked your indices as read-only due to this. Can you check this? ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [June 25, 2019, 5:04am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/7 "2019-06-25T05:04:01Z")
Yes, couple of occasions filesystems were and we deleted the indices file to make more free space. Since then we are facing this issue and first we faced this in May month and last time is 2 weeks back. Please help to fix this issue as we are not seeing any data in kibana dashboards when we delete the indices to make more free space ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [June 25, 2019, 8:41am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/8 "2019-06-25T08:41:40Z")
You need to mark the index as writeable again. ```auto PUT /index-name/_settings { "index.blocks.read_only_allow_delete": null } ``` Alternatively, you can look in index management at the index in cause (.kibana\_task\_manager) and change the setting there using the UI. ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [June 26, 2019, 3:25am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/9 "2019-06-26T03:25:20Z")
> [@Marius\_Dragomir](#): > > PUT /index-name/\_settings { "index.blocks.read\_only\_allow\_delete": null } Hi Marius, Ran the above command to make index writable and got the 404 error as shown below { "error" : { "root\_cause" : [ { "type" : "index\_not\_found\_exception", "reason" : "no such index", "resource.type" : "index\_or\_alias", "resource.id" : "index-name", "index\_uuid" : "_na_", "index" : "index-name" } ], "type" : "index\_not\_found\_exception", "reason" : "no such index", "resource.type" : "index\_or\_alias", "resource.id" : "index-name", "index\_uuid" : "_na_", "index" : "index-name" }, "status" : 404 } Alternatively did not find the option .kibana\_task\_manager in UI index management. Also noticed the health of few indexes are in yellow and showing the below message Index lifecycle error illegal\_argument\_exception: index.lifecycle.rollover\_alias [oldpacketbeat] does not point to index [packetbeat-6.6.1-2019.06.23] As we have index lifecycle policy called "datastream\_policy" where Hotphase rollover is enabled for 7 days and 10 GB and then delete phase enabled for older 5 days from rollover date ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [June 26, 2019, 3:30am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/10 "2019-06-26T03:30:06Z")
I found the "edit settings" tab on particular index in index management section where i changed from "index.blocks.read\_only\_allow\_delete": "true", to "index.blocks.read\_only\_allow\_delete": "false" Can you confirm is above way i did correctly ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [July 24, 2019, 2:50am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/11 "2019-07-24T02:50:28Z")
Hi Marius Dragomir, Can i have an update on this as i need to fix this ASAP.. Thanks ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [July 24, 2019, 2:54am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/12 "2019-07-24T02:54:13Z")
@Marius_Dragomir, Could you please let me how to housekeep the indices as i have created index rollover policy with hot,warm and cold phases for 1 day each and after 7 index will deleted. Here i am facing issue with index alias does not match with rollover policy error Need an urgent help Thanks ---
**Author:** ![ksunil](https://avatars.discourse-cdn.com/v4/letter/k/439d5e/32.png) [@ksunil](https://discuss.elastic.co/u/ksunil)\ **Post date:** [July 31, 2019, 8:16am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/13 "2019-07-31T08:16:18Z")
Hi, As part of housekkeping, all indices in elastic search have been deleted and tried restart elasticsearch and kibana. However kibana is not running and kibana logs shows "503 HTTP" error {"type":"log","@timestamp":"2019-07-31T07:36:51Z","tags":["fatal","root"],"pid":28159,"message":"{ [search\_phase\_execution\_exception] all shards failed :: {"path":"/.kibana/doc/\_count","query":{},"body":"{\"query\":{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"exists\":{\"field\":\"index-pattern\"}},{\"bool\":{\"must\_not\":{\"term\":{\"migrationVersion.index-pattern\":\"6.5.0\"}}}}]}},{\"bool\":{\"must\":[{\"exists\":{\"field\":\"visualization\"}},{\"bool\":{\"must\_not\":{\"term\":{\"migrationVersion.visualization\":\"6.7.2\"}}}}]}}]}}}","statusCode":503,"response":"{\"error\":{\"root\_cause\":,\"type\":\"search\_phase\_execution\_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed\_shards\":},\"status\":503}"}\n at respond (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (\_stream\_readable.js:1103:12)\n at process.\_tickCallback (internal/process/next\_tick.js:63:19)\n status: 503,\n displayName: 'ServiceUnavailable',\n message:\n 'all shards failed: [search\_phase\_execution\_exception] all shards failed',\n path: '/.kibana/doc/\_count',\n query: {},\n body:\n { error:\n { root\_cause: ,\n type: 'search\_phase\_execution\_exception',\n reason: 'all shards failed',\n phase: 'query',\n grouped: true,\n failed\_shards: },\n status: 503 },\n statusCode: 503,\n response:\n '{"error":{"root\_cause":,"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":},"status":503}',\n toString: [Function],\n toJSON: [Function],\n isBoom: true,\n isServer: true,\n data: null,\n output:\n { statusCode: 503,\n payload:\n { message:\n 'all shards failed: [search\_phase\_execution\_exception] all shards failed',\n statusCode: 503,\n error: 'Service Unavailable' },\n headers: {} },\n reformat: [Function],\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable' }"} ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 28, 2019, 8:16am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-in-kibana-canvas/186622/14 "2019-08-28T08:16:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.