Kibana 6.4.2
ElasticSearch 6.4.2
We are getting this error in Kibana having left a dashboard connected overnight
Version: 6.4.2
Build: 18010
Error: Request to Elasticsearch failed: {"error":{"root_cause":,"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":},"status":503}
at http://au-melaqssi02p:5601/bundles/commons.bundle.js:3:1095506
at Function.Promise.try (http://au-melaqssi02p:5601/bundles/commons.bundle.js:3:807416)
at http://au-melaqssi02p:5601/bundles/commons.bundle.js:3:806787
at Array.map ()
at Function.Promise.map (http://au-melaqssi02p:5601/bundles/commons.bundle.js:3:806745)
at callResponseHandlers (http://au-melaqssi02p:5601/bundles/commons.bundle.js:3:1094865)
at http://au-melaqssi02p:5601/bundles/commons.bundle.js:3:1080844
at processQueue (http://au-melaqssi02p:5601/bundles/vendors.bundle.js:211:199684)
at http://au-melaqssi02p:5601/bundles/vendors.bundle.js:211:200647
at Scope.$digest (http://au-melaqssi02p:5601/bundles/vendors.bundle.js:211:210409)
at http://au-melaqssi02p:5601/bundles/vendors.bundle.js:211:212941
at completeOutstandingRequest (http://au-melaqssi02p:5601/bundles/vendors.bundle.js:211:64424)
at http://au-melaqssi02p:5601/bundles/vendors.bundle.js:211:67265
The Elastic log doesn't have any errors, only the following warnings: -
Line 297: [2018-10-31T12:51:26,111][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_jobs_data_2018_10_31t05_00_27][1]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
Line 470: [2018-10-31T13:03:27,187][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_dictionary_errors_2018_10_30t17_12_33][0]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
Line 547: [2018-10-31T13:04:44,745][WARN ][o.e.c.a.s.ShardStateAction] [AU-MELAQSSI02P] unexpected failure while sending request [internal:cluster/shard/started] to [{AU-MELAQSSI02P}{TnFjt3tCRRWY3mW1EngsCw}{BGjzZX9hQxaUZ-tatVOIKA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179398144, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] for shard entry [StartedShardEntry{shardId [[index_integrity_gct_data_2018_10_24t11_44_14][3]], allocationId [4ijX0SqwQjmKTzEDGhcykg], message [after existing recovery]}]
Line 564: [2018-10-31T13:04:44,745][WARN ][o.e.c.a.s.ShardStateAction] [AU-MELAQSSI02P] unexpected failure while sending request [internal:cluster/shard/started] to [{AU-MELAQSSI02P}{TnFjt3tCRRWY3mW1EngsCw}{BGjzZX9hQxaUZ-tatVOIKA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179398144, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] for shard entry [StartedShardEntry{shardId [[index_integrity_gct_data_2018_10_24t11_44_14][0]], allocationId [yv4XJc6MSeuvdbJIiMM91A], message [after existing recovery]}]
Line 625: [2018-10-31T13:05:16,389][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_jobs_errors_2018_10_30t22_00_32][2]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
Line 747: [2018-10-31T13:19:08,787][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_jobs_errors_2018_10_30t22_00_32][2]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
Line 977: [2018-10-31T13:34:50,630][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_integrity_rf_active_delimiters_errors_2018_10_30t22_00_32][0]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
Line 1201: [2018-10-31T13:51:52,536][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_jobs_errors_2018_10_30t22_00_32][0]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
Line 1272: [2018-10-31T13:51:53,272][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_integrity_rf_active_delimiters_data_2018_10_25t13_02_09][3]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
Line 1343: [2018-10-31T13:51:53,288][WARN ][o.e.g.GatewayAllocator$InternalPrimaryShardAllocator] [AU-MELAQSSI02P] [index_integrity_rf_active_delimiters_data_2018_10_25t13_02_09][0]: failed to list shard for shard_started on node [TnFjt3tCRRWY3mW1EngsCw]
There is nothing of interest in the Kibana logs.
Any ideas why this error occurred?
Thanks