# Lost of index upon cluster restart **URL:** \ **Category:** Elasticsearch\ **Created:** [June 17, 2010, 3:51pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024 "2010-06-17T15:51:38Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [June 17, 2010, 3:51pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/1 "2010-06-17T15:51:38Z")
Hi, Yesterday I experienced a strange situation on my ES cluster; I basically lost the last 10 days worth of indexed data after modifying my config and resetting the cluster. I am not sure exactly how this happened, and I haven't seen any particular problem in the logs. I'll describe my setup and what I did and maybe someone 🙂 can help shed some light on what I probably did wrong... I have a 3 nodes cluster: node1 is configured as: - http.enabled : true - node.data : false - index.gateway.type : fs - gateway.type : fs node2 & node3: - http.enabled : false - node.data : true - index.gateway.type : fs - gateway.type : fs I am using the default index storage settings (which should be niofs since I am running on Linux) and the default shards and replicas settings (5 shards and 1 replica). Basically I changed the config file on my 3 nodes to modify my custom stopwords configuration. (and btw, do I need to maintain this stopwords config on both my data nodes and my http node??). I went and did a "bin/service/elasticsearch restart" on my 3 nodes, one after the other. When the cluster was back up, there was a "hole" for the last 10 days of indexed data. The logs do not show any particular problem. What did I do wrong? Thanks, Colin ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 17, 2010, 6:51pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/2 "2010-06-17T18:51:26Z")
I don't see where you define the location of the shared FS drive across those 3 machines... . It is explained here: [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). By default, just so elasticsearch can be simply tested with multi nodes on the same machine, it goes to the work directory, but its local and not shared... . -shay.banon On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant \< [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com)\> wrote: > Hi, > > Yesterday I experienced a strange situation on my ES cluster; I > basically lost the last 10 days worth of indexed data after modifying > my config and resetting the cluster. I am not sure exactly how this > happened, and I haven't seen any particular problem in the logs. I'll > describe my setup and what I did and maybe someone 🙂 can help shed > some light on what I probably did wrong... > > I have a 3 nodes cluster: > > node1 is configured as: > > - http.enabled : true > - node.data : false > - index.gateway.type : fs > - gateway.type : fs > > node2 & node3: > > - http.enabled : false > - node.data : true > - index.gateway.type : fs > - gateway.type : fs > > I am using the default index storage settings (which should be niofs > since I am running on Linux) and the default shards and replicas > settings (5 shards and 1 replica). > > Basically I changed the config file on my 3 nodes to modify my custom > stopwords configuration. (and btw, do I need to maintain this > stopwords config on both my data nodes and my http node??). I went and > did a "bin/service/elasticsearch restart" on my 3 nodes, one after the > other. > > When the cluster was back up, there was a "hole" for the last 10 days > of indexed data. The logs do not show any particular problem. > > What did I do wrong? > > Thanks, > Colin ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [June 17, 2010, 7:25pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/3 "2010-06-17T19:25:31Z")
Ok, obviously, I did not get that and when I tested my config it was multiple nodes on a single host... I will then create a shared fs across my nodes and restart my cluster to pickup the new config. Is there a way for me to restart my cluster without loosing my indexes, in the state it is in now? (with a boggus gateway config). Also, in my config I have both: - index.gateway.type : fs - gateway.type : fs Are both required? All I need is to add this? - index.gateway.fs.location : /my/shared/path Thanks for your help. Colin On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > I don't see where you define the location of the shared FS drive across > those 3 machines... . It is explained > here: [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). > By default, just so elasticsearch can be simply tested with multi nodes on > the same machine, it goes to the work directory, but its local and not > shared... . > -shay.banon > > On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > Hi, > > > > Yesterday I experienced a strange situation on my ES cluster; I > > basically lost the last 10 days worth of indexed data after modifying > > my config and resetting the cluster. I am not sure exactly how this > > happened, and I haven't seen any particular problem in the logs. I'll > > describe my setup and what I did and maybe someone 🙂 can help shed > > some light on what I probably did wrong... > > > > I have a 3 nodes cluster: > > > > node1 is configured as: > > > > - http.enabled : true > > - node.data : false > > - index.gateway.type : fs > > - gateway.type : fs > > > > node2 & node3: > > > > - http.enabled : false > > - node.data : true > > - index.gateway.type : fs > > - gateway.type : fs > > > > I am using the default index storage settings (which should be niofs > > since I am running on Linux) and the default shards and replicas > > settings (5 shards and 1 replica). > > > > Basically I changed the config file on my 3 nodes to modify my custom > > stopwords configuration. (and btw, do I need to maintain this > > stopwords config on both my data nodes and my http node??). I went and > > did a "bin/service/elasticsearch restart" on my 3 nodes, one after the > > other. > > > > When the cluster was back up, there was a "hole" for the last 10 days > > of indexed data. The logs do not show any particular problem. > > > > What did I do wrong? > > > > Thanks, > > Colin ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [June 17, 2010, 8:12pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/4 "2010-06-17T20:12:10Z")
I will finally go with the S3 cloud gateway using config like: gateway: type: cloud cloud: chunk\_size: 1g container: mycontainer index : gateway : type : cloud Now, is the last "index.gateway.type : cloud" config needed or it is completely redundant to the node level gateway config? What I understand from the doc is that the index level gateway will inherit from the node level gateway and index level gateway config is only required to specify exceptions to the node level gateway config. Thanks, Colin On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > Ok, obviously, I did not get that and when I tested my config it was > multiple nodes on a single host... > > I will then create a shared fs across my nodes and restart my cluster > to pickup the new config. Is there a way for me to restart my cluster > without loosing my indexes, in the state it is in now? (with a boggus > gateway config). > > Also, in my config I have both: > > - index.gateway.type : fs > - gateway.type : fs > > Are both required? > > All I need is to add this? > > - index.gateway.fs.location : /my/shared/path > > Thanks for your help. > Colin > > On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > I don't see where you define the location of the shared FS drive across > > those 3 machines... . It is explained > > here: [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). > > By default, just so elasticsearch can be simply tested with multi nodes on > > the same machine, it goes to the work directory, but its local and not > > shared... . > > -shay.banon > > > > On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > Hi, > > > > > > Yesterday I experienced a strange situation on my ES cluster; I > > > basically lost the last 10 days worth of indexed data after modifying > > > my config and resetting the cluster. I am not sure exactly how this > > > happened, and I haven't seen any particular problem in the logs. I'll > > > describe my setup and what I did and maybe someone 🙂 can help shed > > > some light on what I probably did wrong... > > > > > > I have a 3 nodes cluster: > > > > > > node1 is configured as: > > > > > > - http.enabled : true > > > - node.data : false > > > - index.gateway.type : fs > > > - gateway.type : fs > > > > > > node2 & node3: > > > > > > - http.enabled : false > > > - node.data : true > > > - index.gateway.type : fs > > > - gateway.type : fs > > > > > > I am using the default index storage settings (which should be niofs > > > since I am running on Linux) and the default shards and replicas > > > settings (5 shards and 1 replica). > > > > > > Basically I changed the config file on my 3 nodes to modify my custom > > > stopwords configuration. (and btw, do I need to maintain this > > > stopwords config on both my data nodes and my http node??). I went and > > > did a "bin/service/elasticsearch restart" on my 3 nodes, one after the > > > other. > > > > > > When the cluster was back up, there was a "hole" for the last 10 days > > > of indexed data. The logs do not show any particular problem. > > > > > > What did I do wrong? > > > > > > Thanks, > > > Colin ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 17, 2010, 8:53pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/5 "2010-06-17T20:53:28Z")
You don't need to set the index.gateway.type, since it will, as you pointed out, inherit the gateway type. Regarding the first setting of shared fs gateway, you need to set the location on the gateway level: gateway: type: fs fs: location: /mnt/sharedfs You should not set the location of the index gateway (its there, but for really advance use cases). The location on the gateway level is important since that where the cluster meta data will be stored. A disclaimer regarding the cloud gateway, i have hear people having problem with it (caused by jclouds). For 0.9 I am going to write a native AWS cloud version. -shay.banon On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant \< [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com)\> wrote: > I will finally go with the S3 cloud gateway using config like: > > gateway: > type: cloud > cloud: > chunk\_size: 1g > container: mycontainer > index : > gateway : > type : cloud > > Now, is the last "index.gateway.type : cloud" config needed or it is > completely redundant to the node level gateway config? What I > understand from the doc is that the index level gateway will inherit > from the node level gateway and index level gateway config is only > required to specify exceptions to the node level gateway config. > > Thanks, > Colin > > On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > Ok, obviously, I did not get that and when I tested my config it was > > multiple nodes on a single host... > > > > I will then create a shared fs across my nodes and restart my cluster > > to pickup the new config. Is there a way for me to restart my cluster > > without loosing my indexes, in the state it is in now? (with a boggus > > gateway config). > > > > Also, in my config I have both: > > > > - index.gateway.type : fs > > - gateway.type : fs > > > > Are both required? > > > > All I need is to add this? > > > > - index.gateway.fs.location : /my/shared/path > > > > Thanks for your help. > > Colin > > > > On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > I don't see where you define the location of the shared FS drive across > > > those 3 machines... . It is explained > > > here: > > > [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). > > > By default, just so elasticsearch can be simply tested with multi nodes > > > on > > > the same machine, it goes to the work directory, but its local and not > > > shared... . > > > -shay.banon > > > > > > On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > Yesterday I experienced a strange situation on my ES cluster; I > > > > basically lost the last 10 days worth of indexed data after modifying > > > > my config and resetting the cluster. I am not sure exactly how this > > > > happened, and I haven't seen any particular problem in the logs. I'll > > > > describe my setup and what I did and maybe someone 🙂 can help shed > > > > some light on what I probably did wrong... > > > > > > > > I have a 3 nodes cluster: > > > > > > > > node1 is configured as: > > > > > > > > - http.enabled : true > > > > - node.data : false > > > > - index.gateway.type : fs > > > > - gateway.type : fs > > > > > > > > node2 & node3: > > > > > > > > - http.enabled : false > > > > - node.data : true > > > > - index.gateway.type : fs > > > > - gateway.type : fs > > > > > > > > I am using the default index storage settings (which should be niofs > > > > since I am running on Linux) and the default shards and replicas > > > > settings (5 shards and 1 replica). > > > > > > > > Basically I changed the config file on my 3 nodes to modify my custom > > > > stopwords configuration. (and btw, do I need to maintain this > > > > stopwords config on both my data nodes and my http node??). I went and > > > > did a "bin/service/elasticsearch restart" on my 3 nodes, one after the > > > > other. > > > > > > > > When the cluster was back up, there was a "hole" for the last 10 days > > > > of indexed data. The logs do not show any particular problem. > > > > > > > > What did I do wrong? > > > > > > > > Thanks, > > > > Colin ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [June 17, 2010, 9:03pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/6 "2010-06-17T21:03:53Z")
Thanks for your help Shay. Ok, so to play it safe, I'll go with the shared fs gateway for now. Regarding the config change + required cluster restart, since my current cluster gateway config is boggus (using local fs storage for gateway) what would be the best procedure to restart my cluster hopefully without loosing my current indexes? Again, my setup is 1 non-data+http node and 2 data+non-http nodes. Thanks, Colin On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > You don't need to set the index.gateway.type, since it will, as you pointed > out, inherit the gateway type. Regarding the first setting of shared fs > gateway, you need to set the location on the gateway level: > gateway: > type: fs > fs: > location: /mnt/sharedfs > You should not set the location of the index gateway (its there, but for > really advance use cases). The location on the gateway level is important > since that where the cluster meta data will be stored. > A disclaimer regarding the cloud gateway, i have hear people having problem > with it (caused by jclouds). For 0.9 I am going to write a native AWS cloud > version. > -shay.banon > > On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > I will finally go with the S3 cloud gateway using config like: > > > > gateway: > > type: cloud > > cloud: > > chunk\_size: 1g > > container: mycontainer > > index : > > gateway : > > type : cloud > > > > Now, is the last "index.gateway.type : cloud" config needed or it is > > completely redundant to the node level gateway config? What I > > understand from the doc is that the index level gateway will inherit > > from the node level gateway and index level gateway config is only > > required to specify exceptions to the node level gateway config. > > > > Thanks, > > Colin > > > > On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > Ok, obviously, I did not get that and when I tested my config it was > > > multiple nodes on a single host... > > > > > > I will then create a shared fs across my nodes and restart my cluster > > > to pickup the new config. Is there a way for me to restart my cluster > > > without loosing my indexes, in the state it is in now? (with a boggus > > > gateway config). > > > > > > Also, in my config I have both: > > > > > > - index.gateway.type : fs > > > - gateway.type : fs > > > > > > Are both required? > > > > > > All I need is to add this? > > > > > > - index.gateway.fs.location : /my/shared/path > > > > > > Thanks for your help. > > > Colin > > > > > > On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > I don't see where you define the location of the shared FS drive across > > > > those 3 machines... . It is explained > > > > > > > > here: [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). > > > > By default, just so elasticsearch can be simply tested with multi nodes > > > > on > > > > the same machine, it goes to the work directory, but its local and not > > > > shared... . > > > > -shay.banon > > > > > > > > On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant > > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > Yesterday I experienced a strange situation on my ES cluster; I > > > > > basically lost the last 10 days worth of indexed data after modifying > > > > > my config and resetting the cluster. I am not sure exactly how this > > > > > happened, and I haven't seen any particular problem in the logs. I'll > > > > > describe my setup and what I did and maybe someone 🙂 can help shed > > > > > some light on what I probably did wrong... > > > > > > > > > > I have a 3 nodes cluster: > > > > > > > > > > node1 is configured as: > > > > > > > > > > - http.enabled : true > > > > > - node.data : false > > > > > - index.gateway.type : fs > > > > > - gateway.type : fs > > > > > > > > > > node2 & node3: > > > > > > > > > > - http.enabled : false > > > > > - node.data : true > > > > > - index.gateway.type : fs > > > > > - gateway.type : fs > > > > > > > > > > I am using the default index storage settings (which should be niofs > > > > > since I am running on Linux) and the default shards and replicas > > > > > settings (5 shards and 1 replica). > > > > > > > > > > Basically I changed the config file on my 3 nodes to modify my custom > > > > > stopwords configuration. (and btw, do I need to maintain this > > > > > stopwords config on both my data nodes and my http node??). I went and > > > > > did a "bin/service/elasticsearch restart" on my 3 nodes, one after the > > > > > other. > > > > > > > > > > When the cluster was back up, there was a "hole" for the last 10 days > > > > > of indexed data. The logs do not show any particular problem. > > > > > > > > > > What did I do wrong? > > > > > > > > > > Thanks, > > > > > Colin ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 17, 2010, 9:41pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/7 "2010-06-17T21:41:53Z")
Can you share your latest configuration, I got several on this thread... -shay.banon On Fri, Jun 18, 2010 at 12:03 AM, Colin Surprenant \< [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com)\> wrote: > Thanks for your help Shay. > > Ok, so to play it safe, I'll go with the shared fs gateway for now. > > Regarding the config change + required cluster restart, since my > current cluster gateway config is boggus (using local fs storage for > gateway) what would be the best procedure to restart my cluster > hopefully without loosing my current indexes? Again, my setup is 1 > non-data+http node and 2 data+non-http nodes. > > Thanks, > Colin > > On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > You don't need to set the index.gateway.type, since it will, as you > > pointed > > out, inherit the gateway type. Regarding the first setting of shared fs > > gateway, you need to set the location on the gateway level: > > gateway: > > type: fs > > fs: > > location: /mnt/sharedfs > > You should not set the location of the index gateway (its there, but for > > really advance use cases). The location on the gateway level is important > > since that where the cluster meta data will be stored. > > A disclaimer regarding the cloud gateway, i have hear people having > > problem > > with it (caused by jclouds). For 0.9 I am going to write a native AWS > > cloud > > version. > > -shay.banon > > > > On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > I will finally go with the S3 cloud gateway using config like: > > > > > > gateway: > > > type: cloud > > > cloud: > > > chunk\_size: 1g > > > container: mycontainer > > > index : > > > gateway : > > > type : cloud > > > > > > Now, is the last "index.gateway.type : cloud" config needed or it is > > > completely redundant to the node level gateway config? What I > > > understand from the doc is that the index level gateway will inherit > > > from the node level gateway and index level gateway config is only > > > required to specify exceptions to the node level gateway config. > > > > > > Thanks, > > > Colin > > > > > > On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > Ok, obviously, I did not get that and when I tested my config it was > > > > multiple nodes on a single host... > > > > > > > > I will then create a shared fs across my nodes and restart my cluster > > > > to pickup the new config. Is there a way for me to restart my cluster > > > > without loosing my indexes, in the state it is in now? (with a boggus > > > > gateway config). > > > > > > > > Also, in my config I have both: > > > > > > > > - index.gateway.type : fs > > > > - gateway.type : fs > > > > > > > > Are both required? > > > > > > > > All I need is to add this? > > > > > > > > - index.gateway.fs.location : /my/shared/path > > > > > > > > Thanks for your help. > > > > Colin > > > > > > > > On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > I don't see where you define the location of the shared FS drive > > > > > across > > > > > those 3 machines... . It is explained > > > > > > > > > > here: > > > > > [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). > > > > > By default, just so elasticsearch can be simply tested with multi > > > > > nodes > > > > > on > > > > > the same machine, it goes to the work directory, but its local and > > > > > not > > > > > shared... . > > > > > -shay.banon > > > > > > > > > > On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant > > > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Yesterday I experienced a strange situation on my ES cluster; I > > > > > > basically lost the last 10 days worth of indexed data after > > > > > > modifying > > > > > > my config and resetting the cluster. I am not sure exactly how this > > > > > > happened, and I haven't seen any particular problem in the logs. > > > > > > I'll > > > > > > describe my setup and what I did and maybe someone 🙂 can help shed > > > > > > some light on what I probably did wrong... > > > > > > > > > > > > I have a 3 nodes cluster: > > > > > > > > > > > > node1 is configured as: > > > > > > > > > > > > - http.enabled : true > > > > > > - node.data : false > > > > > > - index.gateway.type : fs > > > > > > - gateway.type : fs > > > > > > > > > > > > node2 & node3: > > > > > > > > > > > > - http.enabled : false > > > > > > - node.data : true > > > > > > - index.gateway.type : fs > > > > > > - gateway.type : fs > > > > > > > > > > > > I am using the default index storage settings (which should be niofs > > > > > > since I am running on Linux) and the default shards and replicas > > > > > > settings (5 shards and 1 replica). > > > > > > > > > > > > Basically I changed the config file on my 3 nodes to modify my > > > > > > custom > > > > > > stopwords configuration. (and btw, do I need to maintain this > > > > > > stopwords config on both my data nodes and my http node??). I went > > > > > > and > > > > > > did a "bin/service/elasticsearch restart" on my 3 nodes, one after > > > > > > the > > > > > > other. > > > > > > > > > > > > When the cluster was back up, there was a "hole" for the last 10 > > > > > > days > > > > > > of indexed data. The logs do not show any particular problem. > > > > > > > > > > > > What did I do wrong? > > > > > > > > > > > > Thanks, > > > > > > Colin ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [June 18, 2010, 2:22pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/8 "2010-06-18T14:22:05Z")
I have a 3 nodes clusters, with one http+nondata node and two nonhttp+data nodes. - node 1 config: http : enabled : true node : data : false gateway : type : fs - node 2 & 3 http : enabled : false node : data : true gateway : type : fs So the question is, since this current setup is boggus because the gateway fs path is not shared between the nodes, what is my best course of action to correct the problem without loosing my indexes. Since the data sits on nodes 2 & 3, if I restart node 2 with the new config, wait for it to get the replication back from node 3 then proceed with the restart on node 3, wait for replication from node 2 and then finally restart node 1, would that work? I will either use NFS (I'd rather not) or HDFS (see other thread "cloud gateway") for the gateway. Thanks, Colin On Thu, Jun 17, 2010 at 5:41 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Can you share your latest configuration, I got several on this thread... > -shay.banon > > On Fri, Jun 18, 2010 at 12:03 AM, Colin Surprenant > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > Thanks for your help Shay. > > > > Ok, so to play it safe, I'll go with the shared fs gateway for now. > > > > Regarding the config change + required cluster restart, since my > > current cluster gateway config is boggus (using local fs storage for > > gateway) what would be the best procedure to restart my cluster > > hopefully without loosing my current indexes? Again, my setup is 1 > > non-data+http node and 2 data+non-http nodes. > > > > Thanks, > > Colin > > > > On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > You don't need to set the index.gateway.type, since it will, as you > > > pointed > > > out, inherit the gateway type. Regarding the first setting of shared fs > > > gateway, you need to set the location on the gateway level: > > > gateway: > > > type: fs > > > fs: > > > location: /mnt/sharedfs > > > You should not set the location of the index gateway (its there, but for > > > really advance use cases). The location on the gateway level is > > > important > > > since that where the cluster meta data will be stored. > > > A disclaimer regarding the cloud gateway, i have hear people having > > > problem > > > with it (caused by jclouds). For 0.9 I am going to write a native AWS > > > cloud > > > version. > > > -shay.banon > > > > > > On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > I will finally go with the S3 cloud gateway using config like: > > > > > > > > gateway: > > > > type: cloud > > > > cloud: > > > > chunk\_size: 1g > > > > container: mycontainer > > > > index : > > > > gateway : > > > > type : cloud > > > > > > > > Now, is the last "index.gateway.type : cloud" config needed or it is > > > > completely redundant to the node level gateway config? What I > > > > understand from the doc is that the index level gateway will inherit > > > > from the node level gateway and index level gateway config is only > > > > required to specify exceptions to the node level gateway config. > > > > > > > > Thanks, > > > > Colin > > > > > > > > On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant > > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > > > Ok, obviously, I did not get that and when I tested my config it was > > > > > multiple nodes on a single host... > > > > > > > > > > I will then create a shared fs across my nodes and restart my cluster > > > > > to pickup the new config. Is there a way for me to restart my cluster > > > > > without loosing my indexes, in the state it is in now? (with a boggus > > > > > gateway config). > > > > > > > > > > Also, in my config I have both: > > > > > > > > > > - index.gateway.type : fs > > > > > - gateway.type : fs > > > > > > > > > > Are both required? > > > > > > > > > > All I need is to add this? > > > > > > > > > > - index.gateway.fs.location : /my/shared/path > > > > > > > > > > Thanks for your help. > > > > > Colin > > > > > > > > > > On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > I don't see where you define the location of the shared FS drive > > > > > > across > > > > > > those 3 machines... . It is explained > > > > > > > > > > > > here: [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). > > > > > > By default, just so elasticsearch can be simply tested with multi > > > > > > nodes > > > > > > on > > > > > > the same machine, it goes to the work directory, but its local and > > > > > > not > > > > > > shared... . > > > > > > -shay.banon > > > > > > > > > > > > On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant > > > > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Yesterday I experienced a strange situation on my ES cluster; I > > > > > > > basically lost the last 10 days worth of indexed data after > > > > > > > modifying > > > > > > > my config and resetting the cluster. I am not sure exactly how this > > > > > > > happened, and I haven't seen any particular problem in the logs. > > > > > > > I'll > > > > > > > describe my setup and what I did and maybe someone 🙂 can help shed > > > > > > > some light on what I probably did wrong... > > > > > > > > > > > > > > I have a 3 nodes cluster: > > > > > > > > > > > > > > node1 is configured as: > > > > > > > > > > > > > > - http.enabled : true > > > > > > > - node.data : false > > > > > > > - index.gateway.type : fs > > > > > > > - gateway.type : fs > > > > > > > > > > > > > > node2 & node3: > > > > > > > > > > > > > > - http.enabled : false > > > > > > > - node.data : true > > > > > > > - index.gateway.type : fs > > > > > > > - gateway.type : fs > > > > > > > > > > > > > > I am using the default index storage settings (which should be > > > > > > > niofs > > > > > > > since I am running on Linux) and the default shards and replicas > > > > > > > settings (5 shards and 1 replica). > > > > > > > > > > > > > > Basically I changed the config file on my 3 nodes to modify my > > > > > > > custom > > > > > > > stopwords configuration. (and btw, do I need to maintain this > > > > > > > stopwords config on both my data nodes and my http node??). I went > > > > > > > and > > > > > > > did a "bin/service/elasticsearch restart" on my 3 nodes, one after > > > > > > > the > > > > > > > other. > > > > > > > > > > > > > > When the cluster was back up, there was a "hole" for the last 10 > > > > > > > days > > > > > > > of indexed data. The logs do not show any particular problem. > > > > > > > > > > > > > > What did I do wrong? > > > > > > > > > > > > > > Thanks, > > > > > > > Colin ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 20, 2010, 12:28pm UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/9 "2010-06-20T12:28:58Z")
If you are going to use FS / HDFS, you can use the cluster state API to find out which node is the primary, then copy over the content of the FS gateway (under the work directory) and copy it over. -shay.banon On Fri, Jun 18, 2010 at 5:22 PM, Colin Surprenant \< [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com)\> wrote: > I have a 3 nodes clusters, with one http+nondata node and two > nonhttp+data nodes. > > - node 1 config: > > http : > enabled : true > node : > data : false > gateway : > type : fs > > - node 2 & 3 > > http : > enabled : false > node : > data : true > gateway : > type : fs > > So the question is, since this current setup is boggus because the > gateway fs path is not shared between the nodes, what is my best > course of action to correct the problem without loosing my indexes. > Since the data sits on nodes 2 & 3, if I restart node 2 with the new > config, wait for it to get the replication back from node 3 then > proceed with the restart on node 3, wait for replication from node 2 > and then finally restart node 1, would that work? > > I will either use NFS (I'd rather not) or HDFS (see other thread > "cloud gateway") for the gateway. > > Thanks, > Colin > > On Thu, Jun 17, 2010 at 5:41 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Can you share your latest configuration, I got several on this thread... > > -shay.banon > > > > On Fri, Jun 18, 2010 at 12:03 AM, Colin Surprenant > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > Thanks for your help Shay. > > > > > > Ok, so to play it safe, I'll go with the shared fs gateway for now. > > > > > > Regarding the config change + required cluster restart, since my > > > current cluster gateway config is boggus (using local fs storage for > > > gateway) what would be the best procedure to restart my cluster > > > hopefully without loosing my current indexes? Again, my setup is 1 > > > non-data+http node and 2 data+non-http nodes. > > > > > > Thanks, > > > Colin > > > > > > On Thu, Jun 17, 2010 at 4:53 PM, Shay Banon > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > You don't need to set the index.gateway.type, since it will, as you > > > > pointed > > > > out, inherit the gateway type. Regarding the first setting of shared > > > > fs > > > > gateway, you need to set the location on the gateway level: > > > > gateway: > > > > type: fs > > > > fs: > > > > location: /mnt/sharedfs > > > > You should not set the location of the index gateway (its there, but > > > > for > > > > really advance use cases). The location on the gateway level is > > > > important > > > > since that where the cluster meta data will be stored. > > > > A disclaimer regarding the cloud gateway, i have hear people having > > > > problem > > > > with it (caused by jclouds). For 0.9 I am going to write a native AWS > > > > cloud > > > > version. > > > > -shay.banon > > > > > > > > On Thu, Jun 17, 2010 at 11:12 PM, Colin Surprenant > > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > > > I will finally go with the S3 cloud gateway using config like: > > > > > > > > > > gateway: > > > > > type: cloud > > > > > cloud: > > > > > chunk\_size: 1g > > > > > container: mycontainer > > > > > index : > > > > > gateway : > > > > > type : cloud > > > > > > > > > > Now, is the last "index.gateway.type : cloud" config needed or it is > > > > > completely redundant to the node level gateway config? What I > > > > > understand from the doc is that the index level gateway will inherit > > > > > from the node level gateway and index level gateway config is only > > > > > required to specify exceptions to the node level gateway config. > > > > > > > > > > Thanks, > > > > > Colin > > > > > > > > > > On Thu, Jun 17, 2010 at 3:25 PM, Colin Surprenant > > > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > > > > > Ok, obviously, I did not get that and when I tested my config it > > > > > > was > > > > > > multiple nodes on a single host... > > > > > > > > > > > > I will then create a shared fs across my nodes and restart my > > > > > > cluster > > > > > > to pickup the new config. Is there a way for me to restart my > > > > > > cluster > > > > > > without loosing my indexes, in the state it is in now? (with a > > > > > > boggus > > > > > > gateway config). > > > > > > > > > > > > Also, in my config I have both: > > > > > > > > > > > > - index.gateway.type : fs > > > > > > - gateway.type : fs > > > > > > > > > > > > Are both required? > > > > > > > > > > > > All I need is to add this? > > > > > > > > > > > > - index.gateway.fs.location : /my/shared/path > > > > > > > > > > > > Thanks for your help. > > > > > > Colin > > > > > > > > > > > > On Thu, Jun 17, 2010 at 2:51 PM, Shay Banon > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > I don't see where you define the location of the shared FS drive > > > > > > > across > > > > > > > those 3 machines... . It is explained > > > > > > > > > > > > > > here: > > > > > > > [http://www.elasticsearch.com/docs/elasticsearch/index\_modules/gateway/fs/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/gateway/fs/). > > > > > > > By default, just so elasticsearch can be simply tested with multi > > > > > > > nodes > > > > > > > on > > > > > > > the same machine, it goes to the work directory, but its local and > > > > > > > not > > > > > > > shared... . > > > > > > > -shay.banon > > > > > > > > > > > > > > On Thu, Jun 17, 2010 at 6:51 PM, Colin Surprenant > > > > > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Yesterday I experienced a strange situation on my ES cluster; I > > > > > > > > basically lost the last 10 days worth of indexed data after > > > > > > > > modifying > > > > > > > > my config and resetting the cluster. I am not sure exactly how > > > > > > > > this > > > > > > > > happened, and I haven't seen any particular problem in the logs. > > > > > > > > I'll > > > > > > > > describe my setup and what I did and maybe someone 🙂 can help > > > > > > > > shed > > > > > > > > some light on what I probably did wrong... > > > > > > > > > > > > > > > > I have a 3 nodes cluster: > > > > > > > > > > > > > > > > node1 is configured as: > > > > > > > > > > > > > > > > - http.enabled : true > > > > > > > > - node.data : false > > > > > > > > - index.gateway.type : fs > > > > > > > > - gateway.type : fs > > > > > > > > > > > > > > > > node2 & node3: > > > > > > > > > > > > > > > > - http.enabled : false > > > > > > > > - node.data : true > > > > > > > > - index.gateway.type : fs > > > > > > > > - gateway.type : fs > > > > > > > > > > > > > > > > I am using the default index storage settings (which should be > > > > > > > > niofs > > > > > > > > since I am running on Linux) and the default shards and replicas > > > > > > > > settings (5 shards and 1 replica). > > > > > > > > > > > > > > > > Basically I changed the config file on my 3 nodes to modify my > > > > > > > > custom > > > > > > > > stopwords configuration. (and btw, do I need to maintain this > > > > > > > > stopwords config on both my data nodes and my http node??). I > > > > > > > > went > > > > > > > > and > > > > > > > > did a "bin/service/elasticsearch restart" on my 3 nodes, one > > > > > > > > after > > > > > > > > the > > > > > > > > other. > > > > > > > > > > > > > > > > When the cluster was back up, there was a "hole" for the last 10 > > > > > > > > days > > > > > > > > of indexed data. The logs do not show any particular problem. > > > > > > > > > > > > > > > > What did I do wrong? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Colin ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:23am UTC](https://discuss.elastic.co/t/lost-of-index-upon-cluster-restart/3024/10 "2017-07-06T04:23:10Z")