# Fs gateway snapshots **URL:** \ **Category:** Elasticsearch\ **Created:** [June 25, 2010, 2:38pm UTC](https://discuss.elastic.co/t/fs-gateway-snapshots/3040 "2010-06-25T14:38:40Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Diptamay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diptamay/32/2805_2.png) [@Diptamay](https://discuss.elastic.co/u/Diptamay)\ **Post date:** [June 25, 2010, 2:38pm UTC](https://discuss.elastic.co/t/fs-gateway-snapshots/3040/1 "2010-06-25T14:38:40Z")
The problem: The snapshotting only happens when I shut down the node that I am running and not every 30 secs, as I would expect from the below configuration. Did I configure something incorrectly or am I not understand when the snapshots would take place? The configuration, that I have setup is: gateway: type: fs fs: location: /Users/sanyal/Documents/workspace/hb\_indices/meta index: gateway: type: fs fs: location: /Users/sanyal/Documents/workspace/hb\_indices/snapshot snapshot\_interval : 30 snapshot\_on\_close : true memory: enabled: true store: type: niofs number\_of\_shards : 3 number\_of\_replicas : 2 path: logs: /Users/sanyal/Documents/workspace/logs As a background, I am prototyping ES for use in our in-house CMS application. So right now, ES is setup on only my laptop, which is macbook with 2.26 Ghz core 2 duo with 4GB RAM. I am also using only one node for indexing and searching. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 25, 2010, 6:36pm UTC](https://discuss.elastic.co/t/fs-gateway-snapshots/3040/2 "2010-06-25T18:36:39Z")
Hi, here is the updated configuration that should work: gateway: type: fs fs: location: /path/to/gateawy/location index: gateawy: snapshot\_interval: 30s number\_of\_shards: 3 number\_of\_replicas: 2 path: logs: /path/to/logs * * * Some notes regarding the configuration: 1. You should only define the gateway type to fs on the gateway level, the index level will automatically be FS. Also, the path should only be defined on the gateway level, the index level will reuse it. 2. The snapshot interval is defined on the index.gateway level. Note, by default, a time\_value in elasticsearch is in milliseconds, so you need to define `30s`. Also, I am surprised that you say you did not see snapshotting happen, since the default is 10s. Note, snapshot will only happen if there are changes. 3. I removed the other settings that are the default, like snapshot on close. Note, if you do want to set it, its also on the index.gateway level. 4. You set the number\_of\_shards and number\_of\_replicas. This means that these are the default values now for any index created, unless explicitly specified in the create index API. This applies to all index level settings. -shay.banon On Fri, Jun 25, 2010 at 5:38 PM, diptamay [diptamay@gmail.com](mailto:diptamay@gmail.com) wrote: > The problem: The snapshotting only happens when I shut down the node > that I am running and not every 30 secs, as I would expect from the > below configuration. Did I configure something incorrectly or am I not > understand when the snapshots would take place? > > The configuration, that I have setup is: > > gateway: > type: fs > fs: > location: /Users/sanyal/Documents/workspace/hb\_indices/meta > index: > gateway: > type: fs > fs: > location: /Users/sanyal/Documents/workspace/hb\_indices/snapshot > snapshot\_interval : 30 > snapshot\_on\_close : true > memory: > enabled: true > store: > type: niofs > number\_of\_shards : 3 > number\_of\_replicas : 2 > path: > logs: /Users/sanyal/Documents/workspace/logs > > As a background, I am prototyping ES for use in our in-house CMS > application. So right now, ES is setup on only my laptop, which is > macbook with 2.26 Ghz core 2 duo with 4GB RAM. I am also using only > one node for indexing and searching. ---
**Author:** ![Diptamay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diptamay/32/2805_2.png) [@Diptamay](https://discuss.elastic.co/u/Diptamay)\ **Post date:** [June 25, 2010, 8:07pm UTC](https://discuss.elastic.co/t/fs-gateway-snapshots/3040/3 "2010-06-25T20:07:08Z")
Hi Shay Thanks for the configuration. I see that snapshotting, just keeps updating the segment\_N and translog files. Shouldn't the segment.gen and \*.fdx and \*.fdx files be backed up as well? ## Correct me if I am wrong, so if I stop and start the search server the whole index gets rebuilt from translog? For e.g: ## [15:57:17,257][INFO][node] [Metalhead] {Elasticsearch/0.8.0}[37916]: Initializing ... [15:57:17,261][INFO][plugins] [Metalhead] Loaded [15:57:18,344][INFO][node] [Metalhead] {Elasticsearch/0.8.0}[37916]: Initialized [15:57:18,344][INFO][node] [Metalhead] {Elasticsearch/0.8.0}[37916]: Starting ... [15:57:18,435][INFO][transport] [Metalhead] bound\_address[inet[/0.0.0.0:9300]], publish\_address[inet[/ 169.254.180.96:9300]] [15:57:21,571][INFO][cluster.service] [Metalhead] New Master [Metalhead][aa6c1c96-7b8d-4943-927c-33816a48ee9e][inet[/ 169.254.180.96:9300]], Reason: zen-disco-initial\_connect(master) [15:57:21,611][INFO][discovery] [Metalhead] elasticsearch/aa6c1c96-7b8d-4943-927c-33816a48ee9e [15:57:21,635][INFO][cluster.metadata] [Metalhead] Creating Index [hb\_14], cause [gateway], shards [3]/[2], mappings [audio, article, page] [15:57:21,980][INFO][http] [Metalhead] bound\_address[inet[/0.0.0.0:9200]], publish\_address[inet[/ 169.254.180.96:9200]] [15:57:22,236][INFO][jmx] [Metalhead] bound\_address[service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publish\_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9400/ jmxrmi] If this is so, wouldn't this be a costly thing to do in production with millions of documents? Thanks Diptamay On Jun 25, 2:36 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, here is the updated configuration that should work: > > gateway: > type: fs > fs: > location: /path/to/gateawy/location > index: > gateawy: > snapshot\_interval: 30s > number\_of\_shards: 3 > number\_of\_replicas: 2 > path: > logs: /path/to/logs > > * * * > > Some notes regarding the configuration: > > 1. You should only define the gateway type to fs on the gateway level, the > index level will automatically be FS. Also, the path should only be defined > on the gateway level, the index level will reuse it. > > 2. The snapshot interval is defined on the index.gateway level. Note, by > default, a time\_value in elasticsearch is in milliseconds, so you need to > define `30s`. Also, I am surprised that you say you did not see snapshotting > happen, since the default is 10s. Note, snapshot will only happen if there > are changes. > > 3. I removed the other settings that are the default, like snapshot on > close. Note, if you do want to set it, its also on the index.gateway level. > > 4. You set the number\_of\_shards and number\_of\_replicas. This means that > these are the default values now for any index created, unless explicitly > specified in the create index API. This applies to all index level settings. > > -shay.banon > > On Fri, Jun 25, 2010 at 5:38 PM, diptamay [dipta...@gmail.com](mailto:dipta...@gmail.com) wrote: > > > The problem: The snapshotting only happens when I shut down the node > > that I am running and not every 30 secs, as I would expect from the > > below configuration. Did I configure something incorrectly or am I not > > understand when the snapshots would take place? > > > The configuration, that I have setup is: > > > gateway: > > type: fs > > fs: > > location: /Users/sanyal/Documents/workspace/hb\_indices/meta > > index: > > gateway: > > type: fs > > fs: > > location: /Users/sanyal/Documents/workspace/hb\_indices/snapshot > > snapshot\_interval : 30 > > snapshot\_on\_close : true > > memory: > > enabled: true > > store: > > type: niofs > > number\_of\_shards : 3 > > number\_of\_replicas : 2 > > path: > > logs: /Users/sanyal/Documents/workspace/logs > > > As a background, I am prototyping ES for use in our in-house CMS > > application. So right now, ES is setup on only my laptop, which is > > macbook with 2.26 Ghz core 2 duo with 4GB RAM. I am also using only > > one node for indexing and searching. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 25, 2010, 8:21pm UTC](https://discuss.elastic.co/t/fs-gateway-snapshots/3040/4 "2010-06-25T20:21:33Z")
The translog is there so a flush (on elasticsearch terms, which maps to performing Lucene commit) will not be needed to be performed for each operation. By default, a flush is executed after 5000 docs have been added to the translog, in which case a commit is done, and a new translog gets created. Until then, there are no "new" files in the index, so they don't get snapshotted to the gateway, only the translog. So, to your question, at the upmost, only 5000 docs will need to be reapplied to to a recovered shard from the gateway, not all the changes done, and this is manageable. -shay.banon On Fri, Jun 25, 2010 at 11:07 PM, diptamay [diptamay@gmail.com](mailto:diptamay@gmail.com) wrote: > Hi Shay > > Thanks for the configuration. I see that snapshotting, just keeps > updating the segment\_N and translog files. Shouldn't the segment.gen > and \*.fdx and \*.fdx files be backed up as well? > > ## Correct me if I am wrong, so if I stop and start the search server the whole index gets rebuilt from translog? For e.g: > > ## [15:57:17,257][INFO][node] [Metalhead] {Elasticsearch/0.8.0}[37916]: Initializing ... [15:57:17,261][INFO][plugins] [Metalhead] Loaded [15:57:18,344][INFO][node] [Metalhead] {Elasticsearch/0.8.0}[37916]: Initialized [15:57:18,344][INFO][node] [Metalhead] {Elasticsearch/0.8.0}[37916]: Starting ... [15:57:18,435][INFO][transport] [Metalhead] bound\_address[inet[/0.0.0.0:9300]], publish\_address[inet[/ 169.254.180.96:9300]] [15:57:21,571][INFO][cluster.service] [Metalhead] New Master [Metalhead][aa6c1c96-7b8d-4943-927c-33816a48ee9e][inet[/ 169.254.180.96:9300]], Reason: zen-disco-initial\_connect(master) [15:57:21,611][INFO][discovery] [Metalhead] elasticsearch/aa6c1c96-7b8d-4943-927c-33816a48ee9e [15:57:21,635][INFO][cluster.metadata] [Metalhead] Creating Index [hb\_14], cause [gateway], shards [3]/[2], mappings [audio, article, page] [15:57:21,980][INFO][http] [Metalhead] bound\_address[inet[/0.0.0.0:9200]], publish\_address[inet[/ 169.254.180.96:9200]] [15:57:22,236][INFO][jmx] [Metalhead] bound\_address[service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publish\_address[service:jmx:rmi:///jndi/rmi://169.254.180.96:9400/ jmxrmi] > > If this is so, wouldn't this be a costly thing to do in production > with millions of documents? > > Thanks > Diptamay > > On Jun 25, 2:36 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hi, here is the updated configuration that should work: > > > > gateway: > > type: fs > > fs: > > location: /path/to/gateawy/location > > index: > > gateawy: > > snapshot\_interval: 30s > > number\_of\_shards: 3 > > number\_of\_replicas: 2 > > path: > > logs: /path/to/logs > > > > * * * > > > > Some notes regarding the configuration: > > > > 1. You should only define the gateway type to fs on the gateway level, > > the > > index level will automatically be FS. Also, the path should only be > > defined > > on the gateway level, the index level will reuse it. > > > > 2. The snapshot interval is defined on the index.gateway level. Note, by > > default, a time\_value in elasticsearch is in milliseconds, so you need to > > define `30s`. Also, I am surprised that you say you did not see > > snapshotting > > happen, since the default is 10s. Note, snapshot will only happen if > > there > > are changes. > > > > 3. I removed the other settings that are the default, like snapshot on > > close. Note, if you do want to set it, its also on the index.gateway > > level. > > > > 4. You set the number\_of\_shards and number\_of\_replicas. This means that > > these are the default values now for any index created, unless explicitly > > specified in the create index API. This applies to all index level > > settings. > > > > -shay.banon > > > > On Fri, Jun 25, 2010 at 5:38 PM, diptamay [dipta...@gmail.com](mailto:dipta...@gmail.com) wrote: > > > > > The problem: The snapshotting only happens when I shut down the node > > > that I am running and not every 30 secs, as I would expect from the > > > below configuration. Did I configure something incorrectly or am I not > > > understand when the snapshots would take place? > > > > > The configuration, that I have setup is: > > > > > gateway: > > > type: fs > > > fs: > > > location: /Users/sanyal/Documents/workspace/hb\_indices/meta > > > index: > > > gateway: > > > type: fs > > > fs: > > > location: /Users/sanyal/Documents/workspace/hb\_indices/snapshot > > > snapshot\_interval : 30 > > > snapshot\_on\_close : true > > > memory: > > > enabled: true > > > store: > > > type: niofs > > > number\_of\_shards : 3 > > > number\_of\_replicas : 2 > > > path: > > > logs: /Users/sanyal/Documents/workspace/logs > > > > > As a background, I am prototyping ES for use in our in-house CMS > > > application. So right now, ES is setup on only my laptop, which is > > > macbook with 2.26 Ghz core 2 duo with 4GB RAM. I am also using only > > > one node for indexing and searching. ---
**Author:** ![Diptamay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diptamay/32/2805_2.png) [@Diptamay](https://discuss.elastic.co/u/Diptamay)\ **Post date:** [June 25, 2010, 9:22pm UTC](https://discuss.elastic.co/t/fs-gateway-snapshots/3040/5 "2010-06-25T21:22:40Z")
Hi Shay Thanks for the info. I see the \*.cfs files getting snapshotted, now that I loaded like 100k documents. -Diptamay ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/fs-gateway-snapshots/3040/6 "2017-07-06T04:22:59Z")