All indexes deleted (including .kibana)


(Jerzy Stopa) #1

Hi,

I suddenly lost all data in elasticsearch. It is single node system.
Log shows following:
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.17/cY3e38j6Toq4eNhf-Yegzw] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge/0LtRrh-QSh-XJ_KeG1iqWA] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.watches/1OGO3lR9S8uex_owhm7vOQ] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge_nlschraard_cwc_5/lVHnX8jsS8eYo9aBZTT38A] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_status/k4GWZrCrTKaJzONmekpggA] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.15/b9GmgidcT_-gpZMLuKX2oA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_last_cwc/U1WfmScHQ0W2CmXDWW81RA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-alerts-6/RVCsytE0RqGZykZa4rQTJA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-anomalies-shared/fEzHIMa8Rl-581th-4gTLA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-state/CGUy1zw1QqeHfFlzauLh1w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge_plwrsymnfo_2/sAONFc1yTUK9sXF366pLCA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_cwc_names/PT-xqdU9SleKD540rB_eHQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.17/JwHPt6sLRkifZishIrKEnw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.16/TQ1rH_3kRNiHKx2d8ROX3A] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.14/4-yA3QSRQRueiGAx8Su7Mw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.reporting-2018.05.20/onHOQtdpRQO725kD-C8O7g] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_cwc/rW9oY1PQSBOY4TzbujY1kw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.kibana/y2pQ5J3kSfytHDKvefrkiQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.12/cZGgrd85TDuqn8fCT6_eOw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_other/rYn5oY9_TaazhWNY1vjYQA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.12/EWhAK6VlQhqNDRyZlF6SUw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_mqa/OyCqUwF9SfigMePcj9Oiiw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-notifications/wo2_hR0IT7ObfazbRl9GDA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.13/5WADOFiVRvCaCQiVPgIN7Q] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.13/58AZohfhQRCOdaATVRd08w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.11/ROdbcefuQaKtlHeC_bfXWQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.16/n0dCegnyQy22dH_DP3oOZA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.15/W1cTX0h_T-ewbtsY2_vxwQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.14/_T4Qu7P0RfWfrpzyxaAhwg] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.triggered_watches/0du0cL2pS5qs9keblfbJ-w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.11/3Moi-Jq3Q7SJBr-fZsMQCA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.security-6/HoobTo-YT3ON4vdmWr2OBw] deleting index

No idea what triggered the delete. Any help appreciated.
It is development system and we can get the data back but all kibana visualisations are lost.

BR,
Jerzy


(Christian Dahlqvist) #2

Is it by any chance not secured and open to the internet?

If not, is there anything in the logs?


(Jerzy Stopa) #3

It is not secured node. Or since we upgraded to 6.3 couple of weeks ago it become open and the we didn't address the issue.
You believe it may be intentional?


(Christian Dahlqvist) #4

There have been reports about unsecured clusters that are open to the internet having been targeted, so that is a possibility. Is there anything in the logs around index deletion?


(Jerzy Stopa) #5

about 1.5h before:
[2018-07-17T06:04:23,606][INFO ][o.e.c.m.MetaDataIndexTemplateService] [n524Xf9] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2018-07-17T06:04:26,147][INFO ][o.e.c.m.MetaDataIndexTemplateService] [n524Xf9] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2018-07-17T07:13:35,379][DEBUG][o.e.a.b.TransportShardBulkAction] [sym_edge][3] failed to execute bulk item (index) BulkShardRequest [[sym_edge][3]] containing [index {[sym_edge][machinedata][L9wWp2QBqBZpXlaZUjiS], source[{"timestamp":"2018-07-17T07:13:33","nodeId":"02017968d00a","elapsedTime":0.7,"time":1531811613,"applicationId":"PLWRSYMELIWELL","payload":{"414":"error","412_value_max":3276.8,"412_value_average":3276.8,"412":3276.8,"412_value_min":3276.8}}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [payload.414]

then few parsing errors and then delete index actions.


(Christian Dahlqvist) #6

It does indeed look like someone has deleted the indices. Do you have any snapshot to restore from?


(Jerzy Stopa) #7

Is it possible that system initiated deletion? As response to lack of disk space (we had plenty though) or other event?

I do not have snapshot not any backup. Once again good lesson received.
Luckelly it was mostly leraning system with some visualisation development so nothing crucial is lost.
Some work to re-create visualisations.

Can I get any security with basic X-pack (we use version 6.3).
I plan to go for hosted version for production but for next few months we need to stick to what is available.

I will configure snapshots and some extra backup....


(Christian Dahlqvist) #8

Elasticsearch does not delete data, even if there is a lack of disk space.

The basic license does not include security.


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.