# Exclamation Triangle errors intermittently **URL:** \ **Category:** Kibana\ **Tags:** canvas\ **Created:** [November 26, 2019, 5:10pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552 "2019-11-26T17:10:50Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![kevinpd](https://avatars.discourse-cdn.com/v4/letter/k/49beb7/32.png) [@kevinpd](https://discuss.elastic.co/u/kevinpd)\ **Post date:** [November 26, 2019, 5:10pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/1 "2019-11-26T17:10:50Z")
I have a Canvas display running in Chrome in kiosk mode. It runs 24/7. I have it refreshed every 15 seconds. The Canvas has about 60 expressions that it displays. Some text, a few image reveals, a few lines. The display seldom shows with no exclamation point triangles but where the triangles are varies quite a lot. It seems that pretty much any of the fields may show as triangles at some point. The desktop computer that has the display also runs elasticsearch and kibana locally. The machine is not showing any signs of overwork. Lots of memory and cpu cycles available. How can I diagnose and fix the issue so that I have a clean, error-free, display, regularly? ---
**Author:** ![tims](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tims/32/46750_2.png) [@tims](https://discuss.elastic.co/u/tims)\ **Post date:** [November 26, 2019, 5:28pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/2 "2019-11-26T17:28:32Z")
Hey @kevinpd, since you are running kibana and elasticsearch locally can you see any errors in the log files when the expression fails to execute? If so, can you post the log here and we can take a look. ---
**Author:** ![kevinpd](https://avatars.discourse-cdn.com/v4/letter/k/49beb7/32.png) [@kevinpd](https://discuss.elastic.co/u/kevinpd)\ **Post date:** [November 26, 2019, 5:42pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/3 "2019-11-26T17:42:36Z")
Does this help? I did journalctl -u kibana to get this. Nov 26 08:27:54 kevinpd-linux-02 kibana[4019]: {"type":"error","@timestamp":"2019-11-26T16:27:54Z","tags":["warning","stats-collection"],"pid":4019,"level":"error","error":{"message":"[search\_phase\_execution\_exception] all shards failed","name":"Error","stack":"[search\_phase\_execution\_exception] all shards failed :: {"path":"/.kibana/\_search","query":{"size":10000,"ignore\_unavailable":true,"filter\_path":"hits.hits.\_source.canvas-workpad,-hits.hits.\_source.canvas-workpad.assets"},"body":"{\"query\":{\"bool\":{\"filter\":{\"term\":{\"type\":\"canvas-workpad\"}}}}}","statusCode":503,"response":"{\"error\":{\"root\_cause\":,\"type\":\"search\_phase\_execution\_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed\_shards\":},\"status\":503}"}\n at respond (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (\_stream\_readable.js:1103:12)\n at process.\_tickCallback (internal/process/next\_tick.js:63:19)"},"message":"[search\_phase\_execution\_exception] all shards failed"} Nov 26 08:27:54 kevinpd-linux-02 kibana[4019]: {"type":"log","@timestamp":"2019-11-26T16:27:54Z","tags":["warning","stats-collection"],"pid":4019,"message":"Unable to fetch data from kibana collector"} Nov 26 08:27:54 kevinpd-linux-02 kibana[4019]: {"type":"error","@timestamp":"2019-11-26T16:27:54Z","tags":["warning","stats-collection"],"pid":4019,"level":"error","error":{"message":"[search\_phase\_execution\_exception] all shards failed","name":"Error","stack":"[search\_phase\_execution\_exception] all shards failed :: {"path":"/.kibana/\_search","query":{"ignore\_unavailable":true,"filter\_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":503,"response":"{\"error\":{\"root\_cause\":,\"type\":\"search\_phase\_execution\_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed\_shards\":},\"status\":503}"}\n at respond (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (\_stream\_readable.js:1103:12)\n at process.\_tickCallback (internal/process/next\_tick.js:63:19)"},"message":"[search\_phase\_execution\_exception] all shards failed"} Nov 26 08:27:54 kevinpd-linux-02 kibana[4019]: {"type":"log","@timestamp":"2019-11-26T16:27:54Z","tags":["warning","stats-collection"],"pid":4019,"message":"Unable to fetch data from visualization\_types collector"} Nov 26 08:27:54 kevinpd-linux-02 kibana[4019]: {"type":"error","@timestamp":"2019-11-26T16:27:54Z","tags":["warning","stats-collection"],"pid":4019,"level":"error","error":{"message":"[search\_phase\_execution\_exception] all shards failed","name":"Error","stack":"[search\_phase\_execution\_exception] all shards failed :: {"path":"/.kibana\_task\_manager/\_search","query":{"ignore\_unavailable":true},"body":"{\"sort\":[{\"task.runAt\":\"asc\"},{\"\_id\":\"desc\"}],\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":\"task\"}},{\"bool\":{\"filter\":{\"term\":{\"\_id\":\"oss\_telemetry-vis\_telemetry\"}}}}]}}}","statusCode":503,"response":"{\"error\":{\"root\_cause\":,\"type\":\"search\_phase\_execution\_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed\_shards\":},\"status\":503}"}\n at respond (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (\_stream\_readable.js:1103:12)\n at process.\_tickCallback (internal/process/next\_tick.js:63:19)"},"message":"[search\_phase\_execution\_exception] all shards failed"} Nov 26 08:27:54 kevinpd-linux-02 kibana[4019]: {"type":"log","@timestamp":"2019-11-26T16:27:54Z","tags":["warning","stats-collection"],"pid":4019,"message":"Unable to fetch data from maps collector"} Nov 26 08:27:54 kevinpd-linux-02 kibana[4019]: {"type":"error","@timestamp":"2019-11-26T16:27:54Z","tags":["warning","stats-collection"],"pid":4019,"level":"error","error":{"message":"[search\_phase\_execution\_exception] all shards failed","name":"Error","stack":"[search\_phase\_execution\_exception] all shards failed :: {"path":"/.kibana\_task\_manager/\_search","query":{"ignore\_unavailable":true},"body":"{\"sort\":[{\"task.runAt\":\"asc\"},{\"\_id\":\"desc\"}],\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":\"task\"}},{\"bool\":{\"filter\":{\"term\":{\"\_id\":\"Maps-maps\_telemetry\"}}}}]}}}","statusCode":503,"response":"{\"error\":{\"root\_cause\":,\"type\":\"search\_phase\_execution\_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed\_shards\":},\"status\":503}"}\n at respond (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n ---
**Author:** ![tims](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tims/32/46750_2.png) [@tims](https://discuss.elastic.co/u/tims)\ **Post date:** [November 26, 2019, 5:56pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/4 "2019-11-26T17:56:41Z")
This looks like some sort of configuration or failure issue with elasticsearch. Here is a stackoverflow question that has some troubleshooting tips to maybe try: > ---
**Author:** ![kevinpd](https://avatars.discourse-cdn.com/v4/letter/k/49beb7/32.png) [@kevinpd](https://discuss.elastic.co/u/kevinpd)\ **Post date:** [November 26, 2019, 6:14pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/5 "2019-11-26T18:14:10Z")
Thanks Tim. I don't think that stackoverflow was helpful. I don't have any red shards showing but Canvas still has a few exclamation triangles. One idea in that stackoverflow was to set number\_of\_replicas to 0. I tried that. It doesn't appear to make any difference. Still exclamation triangles. ---
**Author:** ![tims](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tims/32/46750_2.png) [@tims](https://discuss.elastic.co/u/tims)\ **Post date:** [November 26, 2019, 7:39pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/6 "2019-11-26T19:39:21Z")
@kevinpd, how many primary shards do you have? What version of elasticsearch/kibana are you running? Can you post the output of the [Cluster health api](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)? ---
**Author:** ![kevinpd](https://avatars.discourse-cdn.com/v4/letter/k/49beb7/32.png) [@kevinpd](https://discuss.elastic.co/u/kevinpd)\ **Post date:** [November 27, 2019, 4:34pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/7 "2019-11-27T16:34:55Z")
Hi Tim, Cluster Info: http localhost:9200/\_cluster/health HTTP/1.1 200 OK content-encoding: gzip content-length: 229 content-type: application/json; charset=UTF-8 { "active\_primary\_shards": 11, "active\_shards": 11, "active\_shards\_percent\_as\_number": 100.0, "cluster\_name": "elasticsearch", "delayed\_unassigned\_shards": 0, "initializing\_shards": 0, "number\_of\_data\_nodes": 1, "number\_of\_in\_flight\_fetch": 0, "number\_of\_nodes": 1, "number\_of\_pending\_tasks": 0, "relocating\_shards": 0, "status": "green", "task\_max\_waiting\_in\_queue\_millis": 0, "timed\_out": false, "unassigned\_shards": 0 } Version Info: http localhost:9200 HTTP/1.1 200 OK content-encoding: gzip content-length: 311 content-type: application/json; charset=UTF-8 { "cluster\_name": "elasticsearch", "cluster\_uuid": "Ogd1cZYKTzaAWLCQX4Qitw", "name": "\*\*\*\*\*\*", "tagline": "You Know, for Search", "version": { "build\_date": "2019-04-05T22:55:32.697037Z", "build\_flavor": "default", "build\_hash": "b7e28a7", "build\_snapshot": false, "build\_type": "deb", "lucene\_version": "8.0.0", "minimum\_index\_compatibility\_version": "6.0.0-beta1", "minimum\_wire\_compatibility\_version": "6.7.0", "number": "7.0.0" } } # Kibana 7.0.0 ---
**Author:** ![tims](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tims/32/46750_2.png) [@tims](https://discuss.elastic.co/u/tims)\ **Post date:** [December 3, 2019, 7:43pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/8 "2019-12-03T19:43:33Z")
Thanks @kevinpd, there's nothing obvious in your configuration to me at least. Have you tried recreating the issue outside of Canvas? Do you notice any elasticsearch queries failing intermittently elsewhere? You may get some better answers if you post this in the elasticsearch forum, I don't think it is Canvas specific. ---
**Author:** ![kevinpd](https://avatars.discourse-cdn.com/v4/letter/k/49beb7/32.png) [@kevinpd](https://discuss.elastic.co/u/kevinpd)\ **Post date:** [December 3, 2019, 9:17pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/9 "2019-12-03T21:17:13Z")
Thanks Tim. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 31, 2019, 9:26pm UTC](https://discuss.elastic.co/t/exclamation-triangle-errors-intermittently/209552/10 "2019-12-31T21:26:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.