How to resolve Dangling indices error on each Index


(Pinki Mondal) #1

Hi All,

I am getting "can not be imported as a dangling index, as index with same name already exists in cluster metadata" message for each index

Please help how to resolve it


(Thiago Souza) #2

Dangling index is stale index data that the cluster is trying to import.

But it seems that you already have indices with the same name, so the dangling indices can not be imported.

To solve, either delete the on-disk stale index data or delete the existing index in Elasticsearch


(Pinki Mondal) #3

Hi @thiago

I am getting this message for all the indices that are present in ES.
How to delete all the stale index?

How can I find All the stale index?

How to ensure that I already have those index in cluster before deleting stale index because I have very important Information in these indices?

Currently I am running one node.


(Thiago Souza) #4

With a single node only, I am not really sure what you did to get into this situation.

Here is what I suggest you to do. Call GET _cat/indices?v, this will list all the indices that are known by the node. The important part of this response is the uuid column. These uuids should match directory names inside this node's data path under nodes/0/indices. For every directory that is matched from the uuid column in GET _cat/indices?v then it means that it is a known index by the node otherwise it is a dangling index that the node is trying to import.

For every dangling index (i.e. not listed in GET _cat/indices) you could simply move them out of this nodes/0/indices into the same directory structure of a new Elasticsearch node installation. By doing this you will end up with two nodes and then you get to decide which node you want to keep.


(Pinki Mondal) #5

On kibana startup I am getting this:

log [09:28:22.623] [info][status][plugin:timelion@5.4.1] Status changed from uninitialized to green - Ready
log [09:28:22.639] [info][listening] Server running at http://127.0.0.1:5601
log [09:28:22.640] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
log [09:28:22.666] [error][status][plugin:elasticsearch@5.4.1] Status changed from yellow to red - Elasticsearch is still initializing the kibana index.
log [09:28:22.666] [error][status][ui settings] Status changed from yellow to red - Elasticsearch plugin is red


(Thiago Souza) #6

That is a completely different thing from what you initially reported.

Do you have latest Elasticsearch logs?

Also, please, properly format your posts and wrap logs between triple ```


(Pinki Mondal) #7

U were telling me to execute the above query but how to do that because I am unable to start kibana


(Pinki Mondal) #8
[2018-05-08T10:59:25,828][INFO ][o.e.c.s.ClusterService   ] [4kL3f1b] new_master {4kL3f1b}{4kL3f1bESXS-7V2U_d4ITg}{fi1AjZn7SjK_b6b9miiGIg}{10.202.14.181}{10.202.14.181:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-05-08T10:59:25,844][INFO ][o.e.h.n.Netty4HttpServerTransport] [4kL3f1b] publish_address {10.202.14.181:9500}, bound_addresses {10.202.14.181:9500}
[2018-05-08T10:59:25,846][INFO ][o.e.n.Node               ] [4kL3f1b] started
[2018-05-08T10:59:28,729][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[faredge_payments_clone/qGeNwYurTGOXaYFPGsqN6Q]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2018-05-08T10:59:28,729][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[autotune-google/LGrgYD5bSHWDZlbtAz7iFw]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2018-05-08T10:59:28,729][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[.kibana/Aw1tRFGhSAKlyMH4FdlzCQ]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2018-05-08T10:59:28,730][INFO ][o.e.g.GatewayService     ] [4kL3f1b] recovered [189] indices into cluster_state
[2018-05-08T10:59:28,906][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[faredge_payments_clone/qGeNwYurTGOXaYFPGsqN6Q]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2018-05-08T10:59:28,907][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[autotune-google/LGrgYD5bSHWDZlbtAz7iFw]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2018-05-08T10:59:28,907][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[.kibana/Aw1tRFGhSAKlyMH4FdlzCQ]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2018-05-08T10:59:29,099][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[faredge_payments_clone/qGeNwYurTGOXaYFPGsqN6Q]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2018-05-08T10:59:34,114][WARN ][o.e.g.DanglingIndicesState] [4kL3f1b] [[autotune-google/LGrgYD5bSHWDZlbtAz7iFw]] can not be imported as a dangling index, as index with same name already exists in cluster metadata

(Thiago Souza) #9

You can get the output of the GET _cat/indices?v command using a command line tool such as curl. For instance:

$ curl 'localhost:9200/_cat/indices?v'

Post the output of this command, please


(Pinki Mondal) #10
health status index                                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   faredge_messaging_2017-08-29_rejected     XRgbE_vFSjihDT4AnvcUjw   5   1          9            0      180kb          180kb
red    open   jira_work_status_sumit                    Q_Ki_FchRsKucgR-ucTNtw   5   1         15            0       32kb           32kb
red    open   axia_fl_msgin-dd_2017-05-03               eFti5TKeR9u2R7UF_ChHAQ   5   1          2            0     15.6kb         15.6kb
red    open   customerproperties                        8sJnqcIURZueU1ofU9XHqw   5   1          3            0     23.5kb         23.5kb
red    open   jdbc_payment_logs                         xEzRalXjRZm7_eCRndJTuA   5   1       4055            0    893.8kb        893.8kb
red    open   faredge_messaging_2017-06                 6lv0axXrTdiQRd2qxAWEyg   5   1      36494          757     27.2mb         27.2mb
red    open   faredge_messaging_2017-03-02_rejected     fIc29AMHRnyx7bzC27fwhA   5   1       1179            0        1mb            1mb
red    open   beeline_log                               fdGmr-O_QS2_rFf9zdN2cA   5   1         21            0     46.2kb         46.2kb
red    open   jiramgt_preetyp_status                    QMIIp6dDSJyIA2mrcFDTLg   5   1         12            0       43kb           43kb
red    open   faredge_messaging_2017-11-16              zM9eej-1QDmPXPdqYxp9kw   5   1          0            0       160b           160b
red    open   bib_logs_multiple                         HPb4irUQQXyrQBmg1nYAiw   5   1         49            0      218kb          218kb
red    open   fe_payments_data1_2017-08-02              M6oKVtuQS3mw97uBZdPF9Q   5   1       1348            0    491.6kb        491.6kb
red    open   logging_log_19jan                         wR03ePQsS06RnekhHDuGVA   5   1       2379            0      1.4mb          1.4mb
red    open   fe_payments_data1_2017-08-24              Ors-gtliRfyPBbqErRpkRA   5   1         24            0     94.4kb         94.4kb
red    open   faredge_messaging_2017-04-27_rejected     fRruUM3aSluS3wYZTyNKzw   5   1          0            0       320b           320b
red    open   faredge_messaging_2017-05-30_rejected     EahmRMyAQd-5mFfoLKe2CQ   5   1          1            0     44.2kb         44.2kb
red    open   faredge_messaging_2017-11-07              aNBFaoFMQ7uODwtsNcWw8g   5   1          5            0     95.4kb         95.4kb
red    open   customer-transactions                     JOFtIPUJQOiINqlf-FVSbQ   5   1         80            0     40.6kb         40.6kb
red    open   kredit_score                              ulQoe1BsQr2xKnf4Mu08Xg   5   1          1            0      5.6kb          5.6kb
red    open   faredge_payment_instructions_2017-07-07   zjSq8CM1REu_Z8OLc1UkzA   5   1        960            0    355.9kb        355.9kb
red    open   payments                                  phARiOONR1S1g4Z7howIvQ   5   1     180712            0     58.4mb         58.4mb
red    open   faredge_messaging_2017-05-29_rejected     qy4uF9MZQkS55s0aRFzFzw   5   1        129            0    603.2kb        603.2kb
red    open   faredge_paymentinst_2017-07-06            jet1ovezToSJ_zO_L0GTMg   5   1        756            0    211.6kb        211.6kb
red    open   faredge_paymentinsts_2017-06-28           ebzibMC4SgGhqOtnziZtuA   5   1       1007            0    339.2kb        339.2kb
red    open   axia_v3_logs                              qFvn5tFzTSKTIhCgLshYUw   5   1        309            0     51.7kb         51.7kb
red    open   faredge_messaging_2017-05-19_rejected     B_xO42VgT2W1PB8R3sySag   5   1          6            0    274.9kb        274.9kb
red    open   faredge_messaging_2017-11-08_rejected     mYZyu-a9QMy3OUPUT7SOmQ   5   1          5            0      254kb          254kb
red    open   metricbeat-2017.06.20                     ADjEIimiRMat8q22FiPNwA   5   1          0            0       160b           160b
red    open   jira_dummy                                CibBVFNpTtmNpCGJDJSh5w   5   1         15            0     31.1kb         31.1kb
red    open   anomaly-full-data                         TC6HyZiKT3-o6JAhVsRDig   5   1       1129            0    205.8kb        205.8kb
red    open   logging                                   jbw11AT7QIWs41fzv0miPg   5   1     166993            0     45.5mb         45.5mb
red    open   faredge_messaging_2017-03                 SjPYOWUoRd-hV8PC_oc9Ow   5   1      40739            0     18.6mb         18.6mb
red    open   axia_v3_app_logs                          XTVnE6C0S3et8Ui8IVsfPw   5   1      19962            0      1.2mb          1.2mb
red    open   metricbeat-2017.08.30                     QvVtZ6-uRWSzDwsJx9xIIQ   5   1          0            0       320b           320b
red    open   axia_fl_msgin-dd_2017-12-13_rejected      Vbxy5q7vTZWkxt_XtF0PEg   5   1          1            0     12.3kb         12.3kb
red    open   a1                                        QjIutx7oRiecwhAlOUnoBA   5   1       4897            0      2.6mb          2.6mb
red    open   faredge_messaging_2017-11-08              DlzZixBnQVKK6JJwNmuLig   5   1          3            0    143.7kb        143.7kb
red    open   geo_mapping_index                         2cIpSywJQNC5blhCXLYKqw   5   1          2            0     37.7kb         37.7kb
red    open   faredge_messaging_2017-02-28_rejected     Ll3h32WVQB-sCGhuIcRUOA   5   1          1         6149      2.9mb          2.9mb
red    open   finnaxia_error_logs                       uDZVQYB1TjyERr7U6Gip8g   5   1        476            0    283.2kb        283.2kb
red    open   faredge_messaging_2017-04-26_rejected     zd_jEloaQCGgotaWSSFQxQ   5   1          4            0    196.2kb        196.2kb
red    open   axia_fl_msgin-dd_2017-08                  JEcO_qGQQHO-9wnHRkA8XQ   5   1         18            0      256kb          256kb
red    open   bib_application                           uYrXRSlLSbyzfdv_yoQGbg   5   1      15938            0     37.4mb         37.4mb
red    open   jira_19novto20jan_data                    L_vYCRIhTa2BcEHTIosNdA   5   1        331            0    378.3kb        378.3kb
red    open   student_preetypapneja1axia                FS9yTVvCQ4qPLyD2-Zb2aA   5   1          0            0       160b           160b
red    open   faredge_messaging_2017-06-08_rejected     Phb0bdTlQaiJNM5NoV2juA   5   1       1528            0      1.7mb          1.7mb
red    open   faredge_date_configuration                S_nuluUJSWiD4Aohtaik0Q   5   1          0            0       160b           160b

(Thiago Souza) #11

Your index listing is not including the .kibana index at all, which doesn't explains why it can't be imported. Did you manually delete this index before get the listing?

Also all indices are in red state, so there are some serious issues going on with this node that is preventing the cluster to get to a working state so Kibana can work too.

I suggest that you:

  1. Stop the node.
  2. For every uuid in that index listing, move away the directory, with the same uuid name, from outside directory (under Elasticsearch data path) nodes/0/indices.
  3. Start the node and verifies if Kibana can connect to it

(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.