All indexes deleted (including .kibana)

Elastic Stack Elasticsearch
1

Hi,

I suddenly lost all data in elasticsearch. It is single node system.
Log shows following:
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.17/cY3e38j6Toq4eNhf-Yegzw] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge/0LtRrh-QSh-XJ_KeG1iqWA] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.watches/1OGO3lR9S8uex_owhm7vOQ] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge_nlschraard_cwc_5/lVHnX8jsS8eYo9aBZTT38A] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_status/k4GWZrCrTKaJzONmekpggA] deleting index
[2018-07-17T07:44:52,833][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.15/b9GmgidcT_-gpZMLuKX2oA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_last_cwc/U1WfmScHQ0W2CmXDWW81RA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-alerts-6/RVCsytE0RqGZykZa4rQTJA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-anomalies-shared/fEzHIMa8Rl-581th-4gTLA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-state/CGUy1zw1QqeHfFlzauLh1w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_edge_plwrsymnfo_2/sAONFc1yTUK9sXF366pLCA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_cwc_names/PT-xqdU9SleKD540rB_eHQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.17/JwHPt6sLRkifZishIrKEnw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.16/TQ1rH_3kRNiHKx2d8ROX3A] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.14/4-yA3QSRQRueiGAx8Su7Mw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.reporting-2018.05.20/onHOQtdpRQO725kD-C8O7g] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_cwc/rW9oY1PQSBOY4TzbujY1kw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.kibana/y2pQ5J3kSfytHDKvefrkiQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.12/cZGgrd85TDuqn8fCT6_eOw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [sym_other/rYn5oY9_TaazhWNY1vjYQA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.12/EWhAK6VlQhqNDRyZlF6SUw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [delaval_mqa/OyCqUwF9SfigMePcj9Oiiw] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.ml-notifications/wo2_hR0IT7ObfazbRl9GDA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.13/5WADOFiVRvCaCQiVPgIN7Q] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.13/58AZohfhQRCOdaATVRd08w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.11/ROdbcefuQaKtlHeC_bfXWQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.16/n0dCegnyQy22dH_DP3oOZA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.15/W1cTX0h_T-ewbtsY2_vxwQ] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-es-6-2018.07.14/_T4Qu7P0RfWfrpzyxaAhwg] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.triggered_watches/0du0cL2pS5qs9keblfbJ-w] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.monitoring-kibana-6-2018.07.11/3Moi-Jq3Q7SJBr-fZsMQCA] deleting index
[2018-07-17T07:44:52,834][INFO ][o.e.c.m.MetaDataDeleteIndexService] [n524Xf9] [.security-6/HoobTo-YT3ON4vdmWr2OBw] deleting index

No idea what triggered the delete. Any help appreciated.
It is development system and we can get the data back but all kibana visualisations are lost.

BR,
Jerzy

2

Is it by any chance not secured and open to the internet?

If not, is there anything in the logs?

3

It is not secured node. Or since we upgraded to 6.3 couple of weeks ago it become open and the we didn't address the issue.
You believe it may be intentional?

4

There have been reports about unsecured clusters that are open to the internet having been targeted, so that is a possibility. Is there anything in the logs around index deletion?

5

about 1.5h before:
[2018-07-17T06:04:23,606][INFO ][o.e.c.m.MetaDataIndexTemplateService] [n524Xf9] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2018-07-17T06:04:26,147][INFO ][o.e.c.m.MetaDataIndexTemplateService] [n524Xf9] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
[2018-07-17T07:13:35,379][DEBUG][o.e.a.b.TransportShardBulkAction] [sym_edge][3] failed to execute bulk item (index) BulkShardRequest [[sym_edge][3]] containing [index {[sym_edge][machinedata][L9wWp2QBqBZpXlaZUjiS], source[{"timestamp":"2018-07-17T07:13:33","nodeId":"02017968d00a","elapsedTime":0.7,"time":1531811613,"applicationId":"PLWRSYMELIWELL","payload":{"414":"error","412_value_max":3276.8,"412_value_average":3276.8,"412":3276.8,"412_value_min":3276.8}}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [payload.414]

then few parsing errors and then delete index actions.

6

It does indeed look like someone has deleted the indices. Do you have any snapshot to restore from?

7

Is it possible that system initiated deletion? As response to lack of disk space (we had plenty though) or other event?

I do not have snapshot not any backup. Once again good lesson received.
Luckelly it was mostly leraning system with some visualisation development so nothing crucial is lost.
Some work to re-create visualisations.

Can I get any security with basic X-pack (we use version 6.3).
I plan to go for hosted version for production but for next few months we need to stick to what is available.

I will configure snapshots and some extra backup....

8

Elasticsearch does not delete data, even if there is a lack of disk space.

The basic license does not include security.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.