Kibana stuck with "Elasticsearch is still initializing the Monitoring indices" error

[Jeff_I]

I have booted the es servers and restarted kibana, but in the status break down in kibana has one error:

plugin:monitoring@5.6.3 Elasticsearch is still initializing the Monitoring indices

Details:

Nodes es version 5.6.3 xpack installed
Kibana version 5.6.3 xpack installed

End of kibana console

log [18:21:11.663] [error][status][plugin:monitoring@5.6.3] Status changed from yellow to red - Elasticsearch is still initializing the Monitoring indices
log [18:21:39.576] [info][license][xpack] Imported license information from Elasticsearch for [monitoring] cluster: mode: basic | status: active | expiry date: 2018-09-13T16:59:59-07:00

/_xpack yeilds:

{
"build": {
"hash": "948f56d",
"date": "2017-10-06T21:40:52.902Z"
},
"license": {
"uid": "*******************",
"type": "basic",
"mode": "basic",
"status": "active",
"expiry_date_in_millis": 1536883199999
},
"features": {
"graph": {
"description": "Graph Data Exploration for the Elastic Stack",
"available": false,
"enabled": true
},
"ml": {
"description": "Machine Learning for the Elastic Stack",
"available": false,
"enabled": true,
"native_code_info": {
"version": "5.6.3",
"build_hash": "5bf431d29b5bff"
}
},
"monitoring": {
"description": "Monitoring for the Elastic Stack",
"available": true,
"enabled": true
},
"security": {
"description": "Security for the Elastic Stack",
"available": false,
"enabled": true
},
"watcher": {
"description": "Alerting, Notification and Automation for the Elastic Stack",
"available": false,
"enabled": true
}
},
"tagline": "You know, for X"
}

Any suggestions?

[tylersmalley]

What is the response from http://localhost:9200/_cat/indices.

Replace localhost:9200 with the host and port of your Elasticsearch cluster.

[Jeff_I]

green open .monitoring-es-6-2017.10.23 3I1T5mMwRt-D5neTK76mhw 1 1 2284105 17133 3.9gb 1.9gb
green open .monitoring-kibana-6-2017.10.20 LEl6V8SSQNSp0GDf8JQAbw 1 1 2020 0 1.8mb 969.5kb
green open .monitoring-kibana-6-2017.10.25 ayhyTDlCQ-q3XvaLolecWg 1 1 408 0 456.5kb 235kb
green open .monitoring-alerts-6 oYJ3gpNPT0W4-b4Nh7WLNw 1 1 1 0 13.3kb 6.6kb
green open .monitoring-kibana-6-2017.10.22 Abc1JT8wQuCf_MUkhsephg 1 1 12 0 97.5kb 48.7kb
red open .monitoring-es-6-2017.10.24 WCvM7vvQTESR5y-m8SM0ZA 1 1
green open .monitoring-es-6-2017.10.21 JqVrv7BFStOEjrImoTBEGQ 1 1 2279828 21387 3.7gb 1.8gb
red open .monitoring-kibana-6-2017.10.23 erulFuvWQhmm8rvtrqiMBw 1 1
red open .monitoring-kibana-6-2017.10.24 NQhXLW0dSwG4y_UvfP_uvQ 1 1
green open .monitoring-kibana-6-2017.10.21 ol55qnoXS2uqzS1VXNE0jg 1 1 25 0 222.2kb 111.1kb
green open .monitoring-kibana-6-2017.10.27 i2UxtHX0TUGaQD5LSq_X7w 1 1 193 0 611.6kb 305.8kb
green open .monitoring-es-6-2017.10.22 33r43NZXQLOpkHZSWVT_ew 1 1 2293154 21354 3.8gb 1.9gb
green open .monitoring-es-6-2017.10.25 ElUV42WxRJ2vbOV4F4NOvw 1 1 2395543 23936 3.7gb 1.8gb
green open .monitoring-es-6-2017.10.27 Rx9CDITyRFaUTG_dB05hSg 1 1 1597206 21301 2.4gb 1.2gb
green open .monitoring-kibana-6-2017.10.26 CHTAgVQMRJqXow-vmC0UWw 1 1 2363 0 2mb 1mb
green open .monitoring-es-6-2017.10.26 _-ppeTZkTKSpeXBSxqgbKA 1 1 2364152 19630 3.5gb 1.7gb
all other indices that are green and look good.
Is it the red .monitoring indices that are causing the issue? Should I just delete those?

[tylersmalley]

Yeah, I assume there are unassigned shards. You can work to re-assign or simply delete that index.

[Jeff_I]

Yep, that did the trick. Thanks Tyler.