All indexes deleted (including .kibana)

Hi,

I suddenly lost all data in elasticsearch. It is single node system.
Log shows following:
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.17/cY3e38j6Toq4eNhf-Yegzw] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge/0LtRrh-QSh-XJ_KeG1iqWA] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.watches/1OGO3lR9S8uex_owhm7vOQ] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge_nlschraard_cwc_5/lVHnX8jsS8eYo9aBZTT38A] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_status/k4GWZrCrTKaJzONmekpggA] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.15/b9GmgidcT_-gpZMLuKX2oA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_last_cwc/U1WfmScHQ0W2CmXDWW81RA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-alerts-6/RVCsytE0RqGZykZa4rQTJA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-anomalies-shared/fEzHIMa8Rl-581th-4gTLA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-state/CGUy1zw1QqeHfFlzauLh1w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge_plwrsymnfo_2/sAONFc1yTUK9sXF366pLCA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_cwc_names/PT-xqdU9SleKD540rB_eHQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.17/JwHPt6sLRkifZishIrKEnw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.16/TQ1rH_3kRNiHKx2d8ROX3A] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.14/4-yA3QSRQRueiGAx8Su7Mw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.reporting-2018.05.20/onHOQtdpRQO725kD-C8O7g] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_cwc/rW9oY1PQSBOY4TzbujY1kw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.kibana/y2pQ5J3kSfytHDKvefrkiQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.12/cZGgrd85TDuqn8fCT6_eOw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_other/rYn5oY9_TaazhWNY1vjYQA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.12/EWhAK6VlQhqNDRyZlF6SUw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_mqa/OyCqUwF9SfigMePcj9Oiiw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-notifications/wo2_hR0IT7ObfazbRl9GDA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.13/5WADOFiVRvCaCQiVPgIN7Q] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.13/58AZohfhQRCOdaATVRd08w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.11/ROdbcefuQaKtlHeC_bfXWQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.16/n0dCegnyQy22dH_DP3oOZA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.15/W1cTX0h_T-ewbtsY2_vxwQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.14/_T4Qu7P0RfWfrpzyxaAhwg] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.triggered_watches/0du0cL2pS5qs9keblfbJ-w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.11/3Moi-Jq3Q7SJBr-fZsMQCA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.security-6/HoobTo-YT3ON4vdmWr2OBw] deleting index

No idea what triggered the delete. Any help appreciated.
It is development system and we can get the data back but all kibana visualisations are lost.

BR,
Jerzy

Is it by any chance not secured and open to the internet?

If not, is there anything in the logs?

It is not secured node. Or since we upgraded to 6.3 couple of weeks ago it become open and the we didn't address the issue.
You believe it may be intentional?

There have been reports about unsecured clusters that are open to the internet having been targeted, so that is a possibility. Is there anything in the logs around index deletion?

about 1.5h before:
[2018-07-17T06:04:23,606][INFO ][o.e.c.m.MetaDataIndexTemplateService] [n524Xf9] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2018-07-17T06:04:26,147][INFO ][o.e.c.m.MetaDataIndexTemplateService] [n524Xf9] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2018-07-17T07:13:35,379][DEBUG][o.e.a.b.TransportShardBulkAction] [sym_edge][3] failed to execute bulk item (index) BulkShardRequest [[sym_edge][3]] containing [index {[sym_edge][machinedata][L9wWp2QBqBZpXlaZUjiS], source[{"timestamp":"2018-07-17T07:13:33","nodeId":"02017968d00a","elapsedTime":0.7,"time":1531811613,"applicationId":"PLWRSYMELIWELL","payload":{"414":"error","412_value_max":3276.8,"412_value_average":3276.8,"412":3276.8,"412_value_min":3276.8}}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [payload.414]

then few parsing errors and then delete index actions.

It does indeed look like someone has deleted the indices. Do you have any snapshot to restore from?

Is it possible that system initiated deletion? As response to lack of disk space (we had plenty though) or other event?

I do not have snapshot not any backup. Once again good lesson received.
Luckelly it was mostly leraning system with some visualisation development so nothing crucial is lost.
Some work to re-create visualisations.

Can I get any security with basic X-pack (we use version 6.3).
I plan to go for hosted version for production but for next few months we need to stick to what is available.

I will configure snapshots and some extra backup....

Elasticsearch does not delete data, even if there is a lack of disk space.

The basic license does not include security.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.