All Index deleting Error

eogredici · January 16, 2020, 11:56pm

Hi,
i am using elasticSearch 7.3 version. My All index were deleted even though there was no intervention. Even i checked disk usage, memory usage, gc.log but i found errors.
İ attached my elastic log. Could you please chechk this log file?

dadoonet · January 17, 2020, 6:02am

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

Apparently someone or something removed your data. Is your cluster secured?

eogredici · January 17, 2020, 4:49pm

hi dadoonet,

thanks for response, my cluster isn't secured.

I had a similar problem today, but my indexes were not deleted, only closed. But i am using role based access and superuser role only me.
why i cant see the elastic cluster red status from log.

[2020-01-17T16:25:43,045][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] completed closing of indices [aiops_maya-2020.01.17, aiops_maya-2020.01.16, aiops_nokta_sms-2020.01.16, aiops_nokta_sms-2020.01.17, aiops_som-2020.01.17, aiops_saha-2020.01.17, aiops_ivr-2020.01.17, aiops_do-2020.01.16, aiops_saha-2020.01.16, aiops_ivr-2020.01.16, aiops_som-2020.01.16, aiops_do-2020.01.17]
[2020-01-17T16:25:43,999][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_saha-2020.01.16][0], [aiops_do-2020.01.16][0], [aiops_som-2020.01.16][0]] ...]).
[2020-01-17T16:35:33,143][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] opening indices [[aiops_saha-2020.01.16/XiqDu0hcQLOj1aZQrCCVzQ],[aiops_ivr-2020.01.17/yqdensM7RKGdw5PfazPUHA],[aiops_som-2020.01.17/NgbwDoayR2mKvPmFQHnaVw],[aiops_nokta_sms-2020.01.16/9OEML7unT6ePH9Qir7iS1A],[aiops_saha-2020.01.17/sCe3f9RsTBKkl0nfRod5aQ],[aiops_ivr-2020.01.16/jwO6TdCWR2ieYyXXjD496A],[aiops_som-2020.01.16/YetBNfvARretDNtrygq4cA],[aiops_maya-2020.01.17/N5jz3njzS4qWbUKOP2vpJQ],[aiops_maya-2020.01.16/Celk50rbScOW2kvQD--UCw]]
[2020-01-17T16:35:33,677][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_som-2020.01.16][0]] ...]).

dadoonet · January 20, 2020, 10:20am

Please read this about how to format.

eogredici · January 20, 2020, 11:10am

Hi dadoonet,

Sorry for my format. i'm resending log.

[2020-01-17T16:25:43,045][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] completed closing of indices [aiops_maya-2020.01.17, aiops_maya-2020.01.16, aiops_nokta_sms-2020.01.16, aiops_nokta_sms-2020.01.17, aiops_som-2020.01.17, aiops_saha-2020.01.17, aiops_ivr-2020.01.17, aiops_do-2020.01.16, aiops_saha-2020.01.16, aiops_ivr-2020.01.16, aiops_som-2020.01.16, aiops_do-2020.01.17]
    [2020-01-17T16:25:43,999][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_saha-2020.01.16][0], [aiops_do-2020.01.16][0], [aiops_som-2020.01.16][0]] ...]).
    [2020-01-17T16:35:33,143][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] opening indices [[aiops_saha-2020.01.16/XiqDu0hcQLOj1aZQrCCVzQ],[aiops_ivr-2020.01.17/yqdensM7RKGdw5PfazPUHA],[aiops_som-2020.01.17/NgbwDoayR2mKvPmFQHnaVw],[aiops_nokta_sms-2020.01.16/9OEML7unT6ePH9Qir7iS1A],[aiops_saha-2020.01.17/sCe3f9RsTBKkl0nfRod5aQ],[aiops_ivr-2020.01.16/jwO6TdCWR2ieYyXXjD496A],[aiops_som-2020.01.16/YetBNfvARretDNtrygq4cA],[aiops_maya-2020.01.17/N5jz3njzS4qWbUKOP2vpJQ],[aiops_maya-2020.01.16/Celk50rbScOW2kvQD--UCw]]
    [2020-01-17T16:35:33,677][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_som-2020.01.16][0]] ...]).

eogredici · January 20, 2020, 11:13am

Hi again,

I am resending first sample error log with by format.

[wldeploy@tudor logs]$ cat aiops.log
[2020-01-16T03:00:00,735][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [.monitoring-es-7-2020.01.16] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [_doc]
[2020-01-16T03:00:04,300][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [.monitoring-kibana-7-2020.01.16] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0], mappings [_doc]
[2020-01-16T03:00:05,326][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_saha-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T03:00:05,395][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_saha-2020.01.16/vqQKn80LTyq0BEonVWDBcA] create_mapping [doc]
[2020-01-16T03:00:26,325][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_maya-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T03:00:26,398][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_maya-2020.01.16/dN7VxwWGQYCDxQW6pk4sIA] create_mapping [doc]
[2020-01-16T03:00:26,416][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_maya-2020.01.16/dN7VxwWGQYCDxQW6pk4sIA] update_mapping [doc]
[2020-01-16T03:02:04,200][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_som-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T03:02:04,275][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_som-2020.01.16/C-05VfJfSKKjlZSA_7rFyA] create_mapping [doc]
[2020-01-16T03:04:09,806][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_do-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T03:04:09,875][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_do-2020.01.16/_ZqBa-aET1KvJvqozEUGOA] create_mapping [doc]
[2020-01-16T03:04:09,880][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_do-2020.01.16/_ZqBa-aET1KvJvqozEUGOA] update_mapping [doc]
[2020-01-16T03:05:58,017][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_nokta_sms-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T03:05:58,089][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_nokta_sms-2020.01.16/NTs7knDMRqmiK9TCteRxIw] create_mapping [doc]
[2020-01-16T03:10:04,013][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_saha-2020.01.16/vqQKn80LTyq0BEonVWDBcA] update_mapping [doc]
[2020-01-16T03:11:23,184][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_som-2020.01.16/C-05VfJfSKKjlZSA_7rFyA] update_mapping [doc]
[2020-01-16T04:00:00,000][INFO ][o.e.x.m.e.l.LocalExporter] [aiops_master0] cleaning up [2] old indices
[2020-01-16T04:00:00,001][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [.monitoring-kibana-7-2020.01.09/TmBp-YZZSXWmdNngP56lGA] deleting index
[2020-01-16T04:00:00,002][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [.monitoring-es-7-2020.01.09/dA7F6gudTeK3mNMbrNnUEw] deleting index
[2020-01-16T05:11:28,809][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_som-2020.01.16/C-05VfJfSKKjlZSA_7rFyA] update_mapping [doc]
[2020-01-16T09:58:49,622][INFO ][o.e.x.s.a.r.TransportPutRoleAction] [aiops_master0] updated role [crm_monitor]
[2020-01-16T10:09:26,764][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [.kibana_2/ti2lygGiQ4-Xx9MCj_wfzg] update_mapping [_doc]
[2020-01-16T10:21:21,181][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_maya-2019.12.11/YcgViO2LS2-mWRIkE4QgzQ] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_maya-2020.01.15/A6pfhHFrQAC9MUt9nS3CCw] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_tarife-2019.10.31/LBdGy6-OTsKyV7fAmtZJDA] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_saha-2019.12.05/DUReOeR9RA-Gqvjhipw5dw] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_saha-2019.12.27/Jp5EWSdZSL-W66BBAK8Lsw] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_maya-2020.01.01/rpMLjRYSTjSoBNmkmt87Ag] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_maya-2019.11.22/ccPFcq60S5yRYafkPfQbAg] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_maya-2019.11.25/JoUwki67TIq0q7_KwMxwEQ] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_maya-2019.11.23/F1eMztxGRVKgkb4OJtpjIA] deleting index
[2020-01-16T10:21:21,182][INFO ][o.e.c.m.MetaDataDeleteIndexService] [aiops_master0] [aiops_maya-2019.11.19/S-_cHENBRWeSIhzaD11aEA] deleting index
[2020-01-16T10:23:24,538][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_do-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T10:23:24,620][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_do-2020.01.16/mfzH7WxVQC2BYlXxAFt25w] create_mapping [doc]
[2020-01-16T10:23:24,625][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_do-2020.01.16/mfzH7WxVQC2BYlXxAFt25w] update_mapping [doc]
[2020-01-16T10:23:32,587][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_som-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T10:23:32,650][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_som-2020.01.16/YetBNfvARretDNtrygq4cA] create_mapping [doc]
[2020-01-16T10:25:04,176][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_saha-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T10:25:04,224][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_saha-2020.01.16/XiqDu0hcQLOj1aZQrCCVzQ] create_mapping [doc]
[2020-01-16T10:25:04,226][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_saha-2020.01.16/XiqDu0hcQLOj1aZQrCCVzQ] update_mapping [doc]
[2020-01-16T10:25:14,030][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_maya-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T10:25:14,084][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_maya-2020.01.16/Celk50rbScOW2kvQD--UCw] create_mapping [doc]
[2020-01-16T10:25:14,085][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_maya-2020.01.16/Celk50rbScOW2kvQD--UCw] update_mapping [doc]
[2020-01-16T10:27:19,737][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_som-2020.01.16/YetBNfvARretDNtrygq4cA] update_mapping [doc]
[2020-01-16T10:28:34,445][INFO ][o.e.c.m.MetaDataCreateIndexService] [aiops_master0] [aiops_nokta_sms-2020.01.16] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-01-16T10:28:34,494][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_nokta_sms-2020.01.16/9OEML7unT6ePH9Qir7iS1A] create_mapping [doc]
[2020-01-16T10:48:15,447][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] closing indices [aiops_saha-2020.01.16/XiqDu0hcQLOj1aZQrCCVzQ],[aiops_nokta_sms-2020.01.16/9OEML7unT6ePH9Qir7iS1A],[aiops_do-2020.01.16/mfzH7WxVQC2BYlXxAFt25w],[aiops_som-2020.01.16/YetBNfvARretDNtrygq4cA],[aiops_maya-2020.01.16/Celk50rbScOW2kvQD--UCw]
[2020-01-16T10:48:15,867][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] completed closing of indices [aiops_maya-2020.01.16, aiops_nokta_sms-2020.01.16, aiops_do-2020.01.16, aiops_saha-2020.01.16, aiops_som-2020.01.16]
[2020-01-16T10:48:16,129][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_do-2020.01.16][0]] ...]).
[2020-01-16T13:19:01,509][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [.kibana_2/ti2lygGiQ4-Xx9MCj_wfzg] update_mapping [_doc]
[2020-01-16T13:20:26,705][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] opening indices [[aiops_saha-2020.01.16/XiqDu0hcQLOj1aZQrCCVzQ]]
[2020-01-16T13:20:26,767][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_saha-2020.01.16][0]] ...]).
[2020-01-16T13:21:19,715][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] opening indices [[aiops_som-2020.01.16/YetBNfvARretDNtrygq4cA]]
[2020-01-16T13:21:19,813][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_som-2020.01.16][0]] ...]).
[2020-01-16T13:21:30,508][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] opening indices [[aiops_maya-2020.01.16/Celk50rbScOW2kvQD--UCw],[aiops_nokta_sms-2020.01.16/9OEML7unT6ePH9Qir7iS1A]]
[2020-01-16T13:21:30,611][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_nokta_sms-2020.01.16][0]] ...]).
[2020-01-16T13:21:34,733][INFO ][o.e.c.m.MetaDataIndexStateService] [aiops_master0] opening indices [[aiops_do-2020.01.16/mfzH7WxVQC2BYlXxAFt25w]]
[2020-01-16T13:21:34,838][INFO ][o.e.c.r.a.AllocationService] [aiops_master0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[aiops_do-2020.01.16][0]] ...]).
[2020-01-16T13:25:56,043][INFO ][o.e.c.m.MetaDataMappingService] [aiops_master0] [aiops_som-2020.01.16/YetBNfvARretDNtrygq4cA] update_mapping [doc]
[2020-01-16T13:56:12,484][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [aiops_master0] updating index lifecycle policy [Deneme]
[2020-01-16T14:07:38,406][INFO ][o.e.x.s.a.u.TransportPutUserAction] [aiops_master0] updated user [crmoper]

dadoonet · January 21, 2020, 8:37am

Someone, something deleted your indices at 2020-01-16T10:21:21.

Are you using Index Lifecycle Management? Looks like you are:

[2020-01-16T13:56:12,484][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [aiops_master0] updating index lifecycle policy [Deneme]

eogredici · January 21, 2020, 11:31am

Hi dadoonet,

Actually i tried use the index lifecyclemanagment, but i couldn't run this.
If the index life cycle managment deleted the all index , wouldn't I see it in the log

JSON for index lifecycle policy Deneme

{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "set_priority": {
                        "priority": 0
                    }
                }
            }
        }
    }
}

dadoonet · January 30, 2020, 3:15am

Do you have other ILM policies?

system · February 27, 2020, 3:15am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.