# Accident deletion of kibana index but visualization is still intact **URL:** \ **Category:** Kibana\ **Created:** [October 29, 2019, 11:13pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766 "2019-10-29T23:13:33Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [October 29, 2019, 11:13pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/1 "2019-10-29T23:13:33Z")
Hello We accidently applied ILM policy to system indexes and it deleted (?) kibana index we are still able to see saved spaces in visualization but can not create new index pattern and can not make any new changes to existing visualization ( spaces). we see forbidden error Please let know if there is any way to fix forbidden error and move forward with index pattern . we are looking for available options without loosing elk data and visualization spaces. ![2019-10-29_19-12-01](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b651ff39be5ff80705aa8c47dcc16d6d730ca029.png) ![2019-10-29_19-15-54](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bbec8fec9e26a03abbe93b86cebac62c634b2e54.png) Elasticsearch and Kibana are on latest 7.4.X version we see below messages in elastic search node while creating index pattern machine\_memory=12187295744, ml.max\_open\_jobs=20, xpack.installed=true} [SENT\_APPLY\_COMMIT] [2019-10-29T16:50:22,545][INFO][o.e.c.m.MetaDataIndexTemplateService] [cd] adding template [.management-beats] for index patterns [.management-beats] [2019-10-29T16:53:53,604][INFO][o.e.c.m.MetaDataIndexTemplateService] [cdl] adding template [.management-beats] for index patterns [.management-beats] [2019-10-29T17:08:01,007][INFO][o.e.c.m.MetaDataIndexTemplateService] [cdldv] adding template [.management-beats] for index patterns [.management-beats] [2019-10-29T17:08:12,253][INFO][o.e.c.m.MetaDataIndexTemplateService] [cdld] adding template [.management-beats] for index patterns [.management-beats] we also see below error messages ``` at java.lang.Thread.run(Thread.java:830) [?:?] ``` [2019-10-29T18:44:14,381][WARN][o.e.x.m.e.l.LocalExporter] [cdldvcd] unexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring.exporter.ExportException: NodeClosedException[node closed {c}{KKTB8ts4StuMsKmYPbeG4g}{Ooc74dVlRKKgqdv2G2Z6Xw}{cdldm}{11.16.116.223:9300}{ilm}{ml.machine\_memory=12187295744, xpack.installed=true, ml.max\_open\_jobs=20}] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.4.0.jar:7.4.0] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [October 29, 2019, 11:23pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/2 "2019-10-29T23:23:17Z")
we are basically looking how can we fix these errors and move forward since we are able to see data coming to elasticsearch and then to kibana dashboard with no issues. After realizing accidental assigning of ILM to all indixes , i have removed them. at this stage we dont know how many system indexes were deleted. Appreciate your help . ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [October 30, 2019, 12:04am UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/3 "2019-10-30T00:04:48Z")
[2019-10-29T13:34:16,234][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [o-sql-test1/wZYogw3KQ\_aFIOF5AHkRDA] deleting index [2019-10-29T13:34:16,826][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [o-alert/opKA8N\_HQKuCp87Ck-n\_7A] deleting index [2019-10-29T13:34:18,218][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [o-2019.09.11/q-8Csi9OSJ23Pmy6cMEx7g] deleting index [2019-10-29T13:34:18,690][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [o-2019.09.12/G-df9H\_fRo-R2K18cAHyKA] deleting index [2019-10-29T13:44:16,093][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [g-logs-2019.09.25/\_2dTjvP9REucqBvBIELwvQ] deleting index [2019-10-29T13:44:16,309][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [g-logs-2019.09.26/uPd1q-nXTZCgceKASZ68vg] deleting index [2019-10-29T13:44:16,775][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [o-test/xYifV4tQRpWhYivOn3jk\_A] deleting index [2019-10-29T13:44:17,000][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [o-database-metrics/3hEaZSr\_RxGdowQO2dotsg] deleting index [2019-10-29T13:54:16,129][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [g-logs/iGXlFWgAST6zsk00I6rZDw] deleting index [2019-10-29T13:54:16,356][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [g-logs-2019.09.24/\_ckcp8ALQlGwMx1UBE2-aQ] deleting index [2019-10-29T14:04:22,931][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [g-logs-2019.09.27/rtG2YppjSDyCPju\_muJ5qw] deleting index [2019-10-29T14:34:17,583][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [active-session-ora-test/Ygf6lPAJQX25aSO6ueNOTg] deleting index [2019-10-29T14:44:16,084][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [.kibana\_1/j9iZKJDYQ9mZSRmQLVw9rg] deleting index [2019-10-29T14:44:16,229][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [filebeat-7.3.1/Mx\_dgtdISMSIgxXfjbdHMw] deleting index [2019-10-29T14:44:16,398][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [testm-buffer-cache-hit-ratio/8NlmVu4vS0-vShtCNU\_6hQ] deleting index [2019-10-29T14:44:16,582][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [logstash-2019.09.09-000001/VgYLxRRtT9aap5dM3RjWTg] deleting index [2019-10-29T14:44:16,758][INFO][o.e.c.m.MetaDataDeleteIndexService] [server1] [filebeat-7.3.2/oaoLcT9DRzaOq\_CnPgSvOg] deleting index ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [October 30, 2019, 1:06am UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/4 "2019-10-30T01:06:49Z")
while trying to make further admin activities , observed below error [cluster\_block\_exception] index [.security-7] blocked by: [FORBIDDEN/8/index write (api)]; so this says , all admin activities are trying to write on this index which is not allowing. this index settings are as below. i have tied refresh and recovery option from kibana devtools api area { ".security-7" : { "settings" : { "index" : { "lifecycle" : { "name" : "" }, "number\_of\_shards" : "1", "auto\_expand\_replicas" : "0-1", "blocks" : { "read\_only\_allow\_delete" : "false", "read\_only" : "false", "write" : "true" }, "provided\_name" : ".security-7", "format" : "6", "creation\_date" : "1570068759097", "analysis" : { "filter" : { "email" : { "type" : "pattern\_capture", "preserve\_original" : "true", "patterns" : [ "([^@]+)", "(\p{L}+)", "(\d+)", "@(.+)" ] } }, "analyzer" : { "email" : { "filter" : [ "email", "lowercase", "unique" ], "tokenizer" : "uax\_url\_email" } } }, "priority" : "1", "number\_of\_replicas" : "1", "uuid" : "6M6B9Cu-QPWLAIeaNzmS5g", "version" : { "created" : "7030299" } } } } } ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [October 30, 2019, 1:19am UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/5 "2019-10-30T01:19:44Z")
@rashmi could you please share your thoughts on this issue. ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [November 18, 2019, 3:55pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/6 "2019-11-18T15:55:15Z")
Can you check the index settings for the .kibana\_x indices in Index Management? It might be that you just made them read-only if no visualization or dashboards disappeared. ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [November 19, 2019, 6:33pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/7 "2019-11-19T18:33:08Z")
Thanks for your response @Marius_Dragomir , i have checked at , kibana indice was deleted accidentally and no place to check if that was in read only mode. but i tried to make all indices read write mode. what i did was , i exported all visualizations to a test invironment, rebuilt the cluster and imported back. we lost the data but since able to use visualizations. since this is dev environment we can take data loss. ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [November 19, 2019, 6:36pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/8 "2019-11-19T18:36:20Z")
That sounds like the proper way of recovery, I don't think I can offer any more help for that. I will also add an enhancement request so that we add an extra warning in ILM when it would impact system indices. It should help avoid these cases. ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [November 19, 2019, 6:44pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/9 "2019-11-19T18:44:56Z")
Thanks @Marius_Dragomir , please request for enhancement. ILM should not be applied for system indeces. One item i was interested to find was how kibana was able to disaplay visualization when index was removed. It was not in memory since i have restated kibana multiple times. It was able to store visualizations/dashboard information somewhere , i was trying to find if that is second kibana index and use that to create alias of kibana system index. ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [November 20, 2019, 6:07pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/10 "2019-11-20T18:07:40Z")
if you send a "GET \_cat/indices" request in the dev tools it will show all the indices. It should also show if there was a .kibana\_1 or .kibana\_2 index. ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [November 20, 2019, 6:30pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/11 "2019-11-20T18:30:10Z")
Right at that time ( i mean before restore), there was only .kibana\_2 index which was in read write mode but didn't allow commits on visualization. ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [November 20, 2019, 8:37pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/12 "2019-11-20T20:37:01Z")
so this was the setting on the .kibana2 index? "index.blocks.read\_only\_allow\_delete": null ---
**Author:** ![ark516](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@ark516](https://discuss.elastic.co/u/ark516)\ **Post date:** [November 27, 2019, 6:54pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/13 "2019-11-27T18:54:17Z")
yes , that was correct @Marius_Dragomir ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 25, 2019, 6:54pm UTC](https://discuss.elastic.co/t/accident-deletion-of-kibana-index-but-visualization-is-still-intact/205766/14 "2019-12-25T18:54:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.