Hi ,
We have the elastic-search environment with three node cluster .Currently the environment is not stable and dashboard are not loading as expected .
Here is the log data
Node 1
SQ8w]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:41,593][WARN ][o.e.g.DanglingIndicesState] [] [[aspbpm-2019.05.23/j94L9N6FSbmx7kDI54RGKw]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:41,593][WARN ][o.e.g.DanglingIndicesState] [] [[access-2019.04.19/3yyOrT1CQZywz53Th9ZCJA]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:41,593][WARN ][o.e.g.DanglingIndicesState] [] [[pcflog-2019.04.19/0kj4s488QtaPHtmqfhBaaA]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:41,593][WARN ][o.e.g.DanglingIndicesState] [] [[access-2019.04.18/QUIzcmAyQvWj8WRbb0MfCg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:41,593][WARN ][o.e.g.DanglingIndicesState] [] [[zabbix-2019.05.10/tDw1Y0VoTvyl41zAS7SHNg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:41,594][WARN ][o.e.g.DanglingIndicesState] [] [[aspaccess-2019.05.10/fhprgwBpQpumpSZeQoqt5g]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:58:48,578][WARN ][o.e.t.TransportService ] [] Received response for a request that has timed out, sent [41218ms] ago, timed out [11217ms] ago, action [internal:discovery/zen/fd/ping], node [{.**********}{ifryjIltQMKRur4m521LvQ}{a1zc99FHR_epo91Y_ncrCg}{10.190.157.78}{10.190.157.78:9300}], id [9598367]
Node 2
[2019-06-14T01:38:34,208][WARN ][o.e.m.j.JvmGcMonitorService] [.] [gc][65731] overhead, spent [35.5s] collecting in the last [36.2s]
[2019-06-14T01:41:17,009][WARN ][o.e.m.j.JvmGcMonitorService] [.] [gc][old][65859][1262] duration [34.3s], collections [1]/[35.4s], total [34.3s]/[6.3m], memory [9.6gb]->[7.9gb]/[9.9gb], all_pools {[young] [227.7mb]->[8.3mb]/[266.2mb]}{[survivor] [31.8mb]->[0b]/[33.2mb]}{[old] [9.4gb]->[7.9gb]/[9.6gb]}
[2019-06-14T01:41:17,009][WARN ][o.e.m.j.JvmGcMonitorService] [.] [gc][65859] overhead, spent [34.4s] collecting in the last [35.4s]
[2019-06-14T01:56:03,297][WARN ][o.e.m.j.JvmGcMonitorService] [.] [gc][old][66702][1293] duration [36.2s], collections [1]/[36.7s], total [36.2s]/[6.9m], memory [9.8gb]->[8gb]/[9.9gb], all_pools {[young] [146.9mb]->[2.2mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old] [9.6gb]->[8gb]/[9.6gb]}
[2019-06-14T01:56:03,297][WARN ][o.e.m.j.JvmGcMonitorService] [.] [gc][66702] overhead, spent [36.3s] collecting in the last [36.7s]
Node 3
[2019-06-14T00:05:40,344][WARN ][o.e.g.DanglingIndicesState] [.] [[access-2019.05.11/wEVDWI99TJq3xKt1nZM52g]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:40,344][WARN ][o.e.g.DanglingIndicesState] [.] [[bpm-2019.05.23/cwKEURoASxaHpo31JTQ49A]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:40,344][WARN ][o.e.g.DanglingIndicesState] [.] [[pcflog-2019.04.19/0kj4s488QtaPHtmqfhBaaA]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:40,344][WARN ][o.e.g.DanglingIndicesState] [.] [[access-2019.04.18/QUIzcmAyQvWj8WRbb0MfCg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:40,344][WARN ][o.e.g.DanglingIndicesState] [.] [[zabbix-2019.05.10/tDw1Y0VoTvyl41zAS7SHNg]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2019-06-14T00:05:40,344][WARN ][o.e.g.DanglingIndicesState] [.] [[aspaccess-2019.05.10/fhprgwBpQpumpSZeQoqt5g]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
please find the cluster data
"cluster_name" : "*******",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 9779,
"active_shards" : 19541,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 257,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 98.70188907970501
The application is not available for more than week . Any one help on this issue is much appreciated .