# Lost template when simulating a disaster scenario
**URL:** https://discuss.elastic.co/t/lost-template-when-simulating-a-disaster-scenario/18711
**Category:** Elasticsearch
**Created:** [July 16, 2014, 8:57pm UTC](https://discuss.elastic.co/t/lost-template-when-simulating-a-disaster-scenario/18711 "2014-07-16T20:57:32Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@Andy\_Walker](https://discuss.elastic.co/u/Andy_Walker)
#### Post date: [July 16, 2014, 8:57pm UTC](https://discuss.elastic.co/t/lost-template-when-simulating-a-disaster-scenario/18711/1 "2014-07-16T20:57:32Z")
Hi all!
First, I'd like to thank the creators of ES, the contributors, and the
large community that surrounds it. I've gotten all sorts of great
information from this list, the IRC channel, and all across the web. But,
onto my scenario/question...
Long story (question below):
Yesterday, my task was to simulate a disaster scenario with my
Elasticsearch cluster, just to see how easy it would be to pull through the
mess. At the time, I was working with a single cluster, and three total
nodes. Two nodes were active when I started, and one was a node that was
completely fresh, didn't have the ES service running on it yet, and had
never yet joined this (or any) cluster. All of the indexes I care about
have shards with 1 replica. I started by ungracefully shutting down the
first of the two nodes, and letting the second node take all of the
activity. Everything was good so far. Obviously no data loss... onto step
two. I brought the first node back up. I then wanted to simulate the
scenario where the second node couldn't handle the extra work of
rebalancing shards as well as handling most or all indexing/query traffic,
so I ungracefully shut down elasticsearch on the second node as well. At
this point, obviously, the cluster health was red, and a good portion of my
data was missing. Feeling confident that I could still recover from this
situation, I shut down elasticsearch on the first node, effectively
shutting off the cluster completely across the board. Then I started ES on
my third node (the fresh one), followed by starting ES on the first and
second nodes as well. Much to my delight, all of the rebalancing took
place, and all of my indexes and shards seemed to survive the storm just
fine. I didn't get to learn any lessons about how to recover from that sort
of disaster, because apparently it wasn't disastrous enough! HOWEVER...
fast-forward to today... I have some daily indexes, and their mappings are
set by a template that was created via the API (not via a config file), and
today I noticed that the field mappings were wrong. I checked for the
template via the API, and to my dismay, it was nowhere to be found.
Question:
Where/how are templates created via the rest API stored? Are they persisted
to disk in any way, or are they just a bit of information that's shared
amongst all of the nodes in a cluster, such that if all nodes were to be
shut down at the same time, the template would be lost? How about in a
split brain situation, which I introduced in my scenario, to an extent. If
a new master comes in and doesn't have knowledge of any custom templates,
but then another node -- which previously had those templates defined --
joins the cluster, does the fresh master win, therefore blowing away the
template?
I'd be more than happy to set up some more test scenarios, and try to
reproduce this, if it would help... but if it's a known issue, or if it
just works that way by design, then I would rather not spend the time.
Thoughts?
Thanks!
Andy Walker
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4ff9593b-80a4-4eee-953d-08d287a7b683%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4ff9593b-80a4-4eee-953d-08d287a7b683%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:15am UTC](https://discuss.elastic.co/t/lost-template-when-simulating-a-disaster-scenario/18711/2 "2017-07-06T01:15:19Z")