# Indices lost after system bounce - MissingIndexException **URL:** \ **Category:** Elasticsearch\ **Created:** [October 26, 2010, 7:56pm UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481 "2010-10-26T19:56:58Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Gideon\_Kaplan](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@Gideon\_Kaplan](https://discuss.elastic.co/u/Gideon_Kaplan)\ **Post date:** [October 26, 2010, 7:56pm UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481/1 "2010-10-26T19:56:58Z")
Hello folks I'm running a mac osx 10.6.4 and we are using elastic search {elasticsearch/0.9.0}[211]: started We are programmatically building 2 indexes with (sub) types via groovy client (and the server has been started separately) Everything is hunky dory while the system is running I can shut it the terminal window down with ctrl-c or passing the shutdown all curl command and restart using elasticsearch -f The problem is when I bounce my system and restart the server it indexes are missing I believe the gateway is set to FS, but the store is set to memory After bounce I do see some stuff in my work directory /usr/local/elasticsearch-0.9.0/work/Ticketfly-1/nodes/0/indices/ backstage/1/index GIDEON-KAPLANs-MacBook-Pro:index gideon$ ls \_0.cfs \_0\_1.del segments.gen segments\_2 But it's looks like it's missing the indices directory in the gateway fs store /tmp/elasticsearch/data/cluster after system bounce (the below dir is recreated when I rebuild the indexes programatically) /tmp/elasticsearch/data/cluster/Ticketfly-1/indices This is before rebuilding GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-0 { "meta-data" : { "indices" : { } } }GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ pwd /tmp/elasticsearch/data/cluster/Ticketfly-1/metadata This is after GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-5 (metadata-5 is the only file there) { "meta-data" : { "indices" : { "inventory" : { "settings" : { "index.number\_of\_shards" : "5", "index.number\_of\_replicas" : "4" }, "mappings" : [ { ..... I do see them on the file system though, maybe I'm just doing something wrong or is there a way to tell it to rediscover the existing indexes on startup or not delete them on shutdown (what I think it's doing) Does one have to set the store to fs in the yml file? Sorry if this is a dumb question, I'm a bit new at using this tech my yml file name: SearchDevelopmentInstance cluster: name: Ticketfly-1 node: data: true http: enabled: true network: #bind\_host: 0.0.0.0 #publish\_host: _eth1_ host: 127.0.0.1 gateway: type: fs fs: location: /tmp/elasticsearch/data/cluster index : number\_of\_shards : 5 number\_of\_replicas : 4 analysis : analyzer : standard : type : standard store: type: memory memory: cache\_size: 100m buffer\_size: 10k #transport: # tcp: # port: 9300 startup after bounce GIDEON-KAPLANs-MacBook-Pro:bin gideon$ elasticsearch -f [11:41:56,616][INFO][node] [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: initializing ... [11:41:56,635][INFO][plugins] [SearchDevelopmentInstance] loaded [] [11:41:58,224][DEBUG][cache.memory] [SearchDevelopmentInstance] using bytebuffer cache with buffer\_size [100kb], cache\_size [200mb], direct [true], warm\_cache [false] [11:41:58,272][DEBUG][threadpool.cached] [SearchDevelopmentInstance] Initializing cached thread pool with keep\_alive[1m], scheduled\_size[20] [11:41:58,317][DEBUG][discovery.zen.ping.multicast] [SearchDevelopmentInstance] using group [224.2.2.4], with port [54328], ttl [3], and address [null] [11:41:58,323][DEBUG][discovery.zen.ping.unicast] [SearchDevelopmentInstance] using initial hosts [] [11:41:58,333][DEBUG][discovery.zen] [SearchDevelopmentInstance] using initial\_ping\_timeout [3s] [11:41:58,335][DEBUG][discovery.zen.fd] [SearchDevelopmentInstance] [master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [11:41:58,342][DEBUG][discovery.zen.fd] [SearchDevelopmentInstance] [node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [11:41:58,410][DEBUG][env] [SearchDevelopmentInstance] using node location [/usr/local/ elasticsearch-0.9.0/work/Ticketfly-1/nodes/0] [11:41:58,504][DEBUG][monitor.memory.alpha] [SearchDevelopmentInstance] interval [500ms], upper\_memory\_threshold [0.95], lower\_memory\_threshold [0.8], translog\_number\_of\_operations\_threshold [5000] [11:41:58,536][DEBUG][monitor.network] [SearchDevelopmentInstance] net\_info host [GIDEON-KAPLANs-MacBook-Pro.local] en1 display\_name [en1] address [/10.59.15.40] [/ fe80:0:0:0:cabc:c8ff:fed9:a56%5] mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual [false] en0 display\_name [en0] address [/10.56.10.139] [/ fe80:0:0:0:cabc:c8ff:fe96:9069%4] mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual [false] lo0 display\_name [lo0] address [/127.0.0.1] [/fe80:0:0:0:0:0:0:1%1] [/ 0:0:0:0:0:0:0:1] mtu [16384] multicast [true] ptp [false] loopback [true] up [true] virtual [false] [11:41:58,626][DEBUG][indices.recovery.throttler] [SearchDevelopmentInstance] concurrent\_recoveries [4], concurrent\_streams [4] interval [100ms] [11:41:58,631][DEBUG][indices.memory] [SearchDevelopmentInstance] using index\_buffer\_size [406.2mb], with min\_shard\_index\_buffer\_size [4mb] [11:41:58,636][DEBUG][gateway.fs] [SearchDevelopmentInstance] Latest metadata found at index [-1] [11:41:58,637][INFO][node] [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: initialized [11:41:58,637][INFO][node] [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: starting ... [11:41:58,698][DEBUG][netty.channel.socket.nio.NioProviderMetadata] Using the autodetected NIO constraint level: 0 [11:41:58,740][DEBUG][transport.netty] [SearchDevelopmentInstance] Bound to address [/127.0.0.1:9300] [11:41:58,741][INFO][transport] [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9300]}, publish\_address {inet[/127.0.0.1:9300]} [11:42:01,865][DEBUG][discovery.zen] [SearchDevelopmentInstance] ping responses: {none} [11:42:01,870][DEBUG][cluster.service] [SearchDevelopmentInstance] processing [zen-disco-join (elected\_as\_master)]: execute [11:42:01,871][DEBUG][cluster.service] [SearchDevelopmentInstance] cluster state updated, version [1], source [zen-disco-join (elected\_as\_master)] [11:42:01,872][INFO][cluster.service] [SearchDevelopmentInstance] new\_master [SearchDevelopmentInstance] [3d1d41e1-4846-4024-8738-230fbd6927b0][inet[/127.0.0.1:9300]], reason: zen-disco-join (elected\_as\_master) [11:42:02,014][DEBUG][transport.netty] [SearchDevelopmentInstance] Connected to node [[SearchDevelopmentInstance][3d1d41e1-4846-4024-8738-230fbd6927b0] [inet[/127.0.0.1:9300]]] [11:42:02,016][DEBUG][cluster.service] [SearchDevelopmentInstance] processing [zen-disco-join (elected\_as\_master)]: done applying updated cluster\_state [11:42:02,016][INFO][discovery] [SearchDevelopmentInstance] Ticketfly-1/3d1d41e1-4846-4024-8738-230fbd6927b0 [11:42:02,016][DEBUG][gateway] [SearchDevelopmentInstance] reading state from gateway fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... [11:42:02,017][DEBUG][gateway] [SearchDevelopmentInstance] read state from gateway fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], took 0s [11:42:02,017][DEBUG][gateway] [SearchDevelopmentInstance] no state read from gateway [11:42:02,018][DEBUG][cluster.service] [SearchDevelopmentInstance] processing [gateway (marked as read, reason=no state)]: execute [11:42:02,019][DEBUG][cluster.service] [SearchDevelopmentInstance] cluster state updated, version [2], source [gateway (marked as read, reason=no state)] [11:42:02,020][DEBUG][gateway] [SearchDevelopmentInstance] writing to gateway fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... [11:42:02,020][DEBUG][cluster.service] [SearchDevelopmentInstance] processing [gateway (marked as read, reason=no state)]: done applying updated cluster\_state [11:42:02,023][INFO][http] [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9200]}, publish\_address {inet[/127.0.0.1:9200]} [11:42:02,172][DEBUG][gateway] [SearchDevelopmentInstance] wrote to gateway fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], took 152ms [11:42:02,236][INFO][jmx] [SearchDevelopmentInstance] bound\_address {service:jmx:rmi:///jndi/ rmi://:9400/jmxrmi}, publish\_address {service:jmx:rmi:///jndi/rmi:// 127.0.0.1:9400/jmxrmi} [11:42:02,236][INFO][node] [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: started [11:42:12,016][DEBUG][cluster.service] [SearchDevelopmentInstance] processing [routing-table-updater]: execute [11:42:12,018][DEBUG][cluster.service] [SearchDevelopmentInstance] processing [routing-table-updater]: no change in cluster\_state query to get a document I know was there before shutdown GIDEON-KAPLANs-MacBook-Pro:~ gideon$ curl -XGET '[http://localhost:9200/](http://localhost:9200/) backstage/event/11637' {"error":"IndexMissingException[[backstage] missing]"}GIDEON-KAPLANs- MacBook-Pro: ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 26, 2010, 9:18pm UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481/2 "2010-10-26T21:18:45Z")
Hi, First, can you try and use the latest version (0.12)? It will be simpler to support this. You have to use the shared gateway (fs) if you are going to store the index in memory. If not, I recommend starting with the loca gateway (which is the default out of the box in 0.12). The index local storage (not gateway) defaults to file system, so you can just remove the store settings in the index. A few points: 1. If you are going to store the index in memory, make sure you have enough of it. 2. File system index can be pretty fast, make sure you really need the index to be in memory for the extra juice. Also, post shutdown, can you check if the indices are listed in the metadata file? -shay.banon On Tue, Oct 26, 2010 at 9:56 PM, Gideon Kaplan [gideonkaplan@gmail.com](mailto:gideonkaplan@gmail.com)wrote: > Hello folks > I'm running a mac osx 10.6.4 and we are using Elasticsearch > {elasticsearch/0.9.0}[211]: started > > We are programmatically building 2 indexes with (sub) types via groovy > client (and the server has been started separately) > Everything is hunky dory while the system is running > I can shut it the terminal window down with ctrl-c or passing the > shutdown all curl command and restart using elasticsearch -f > > The problem is when I bounce my system and restart the server it > indexes are missing > I believe the gateway is set to FS, but the store is set to memory > > After bounce I do see some stuff in my work directory > /usr/local/elasticsearch-0.9.0/work/Ticketfly-1/nodes/0/indices/ > backstage/1/index > GIDEON-KAPLANs-MacBook-Pro:index gideon$ ls > \_0.cfs \_0\_1.del segments.gen segments\_2 > > But it's looks like it's missing the indices directory in the gateway > fs store /tmp/elasticsearch/data/cluster > after system bounce (the below dir is recreated when I rebuild the > indexes programatically) > /tmp/elasticsearch/data/cluster/Ticketfly-1/indices > > This is before rebuilding > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-0 > { > "meta-data" : { > "indices" : { > } > } > }GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ pwd > /tmp/elasticsearch/data/cluster/Ticketfly-1/metadata > > This is after > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-5 (metadata-5 > is the only file there) > { > "meta-data" : { > "indices" : { > "inventory" : { > "settings" : { > "index.number\_of\_shards" : "5", > "index.number\_of\_replicas" : "4" > }, > "mappings" : [ { > ..... > > I do see them on the file system though, maybe I'm just doing > something wrong or is there a way to tell it to rediscover the > existing indexes on startup or not delete them on shutdown (what I > think it's doing) > > Does one have to set the store to fs in the yml file? > > Sorry if this is a dumb question, I'm a bit new at using this tech > > my yml file > name: SearchDevelopmentInstance > > cluster: > name: Ticketfly-1 > > node: > data: true > > http: > enabled: true > > network: > #bind\_host: 0.0.0.0 > #publish\_host: _eth1_ > host: 127.0.0.1 > > gateway: > type: fs > fs: > location: /tmp/elasticsearch/data/cluster > index : > number\_of\_shards : 5 > number\_of\_replicas : 4 > analysis : > analyzer : > standard : > type : standard > > store: > type: memory > memory: > cache\_size: 100m > buffer\_size: 10k > > #transport: > > # tcp: > > # port: 9300 > > startup after bounce > GIDEON-KAPLANs-MacBook-Pro:bin gideon$ elasticsearch -f > [11:41:56,616][INFO][node] > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: > initializing ... > [11:41:56,635][INFO][plugins] > [SearchDevelopmentInstance] loaded > [11:41:58,224][DEBUG][cache.memory] > [SearchDevelopmentInstance] using bytebuffer cache with buffer\_size > [100kb], cache\_size [200mb], direct [true], warm\_cache [false] > [11:41:58,272][DEBUG][threadpool.cached] > [SearchDevelopmentInstance] Initializing cached thread pool with > keep\_alive[1m], scheduled\_size[20] > [11:41:58,317][DEBUG][discovery.zen.ping.multicast] > [SearchDevelopmentInstance] using group [224.2.2.4], with port > [54328], ttl [3], and address [null] > [11:41:58,323][DEBUG][discovery.zen.ping.unicast] > [SearchDevelopmentInstance] using initial hosts > [11:41:58,333][DEBUG][discovery.zen] > [SearchDevelopmentInstance] using initial\_ping\_timeout [3s] > [11:41:58,335][DEBUG][discovery.zen.fd] > [SearchDevelopmentInstance] [master] uses ping\_interval [1s], > ping\_timeout [30s], ping\_retries [3] > [11:41:58,342][DEBUG][discovery.zen.fd] > [SearchDevelopmentInstance] [node] uses ping\_interval [1s], > ping\_timeout [30s], ping\_retries [3] > [11:41:58,410][DEBUG][env] > [SearchDevelopmentInstance] using node location [/usr/local/ > elasticsearch-0.9.0/work/Ticketfly-1/nodes/0] > [11:41:58,504][DEBUG][monitor.memory.alpha] > [SearchDevelopmentInstance] interval [500ms], upper\_memory\_threshold > [0.95], lower\_memory\_threshold [0.8], > translog\_number\_of\_operations\_threshold [5000] > [11:41:58,536][DEBUG][monitor.network] > [SearchDevelopmentInstance] net\_info > host [GIDEON-KAPLANs-MacBook-Pro.local] > en1 display\_name [en1] > address [/10.59.15.40] [/ > fe80:0:0:0:cabc:c8ff:fed9:a56%5] > mtu [1500] multicast [true] ptp [false] loopback > [false] up [true] virtual [false] > en0 display\_name [en0] > address [/10.56.10.139] [/ > fe80:0:0:0:cabc:c8ff:fe96:9069%4] > mtu [1500] multicast [true] ptp [false] loopback > [false] up [true] virtual [false] > lo0 display\_name [lo0] > address [/127.0.0.1] [/fe80:0:0:0:0:0:0:1%1] [/ > 0:0:0:0:0:0:0:1] > mtu [16384] multicast [true] ptp [false] loopback > [true] up [true] virtual [false] > > [11:41:58,626][DEBUG][indices.recovery.throttler] > [SearchDevelopmentInstance] concurrent\_recoveries [4], > concurrent\_streams [4] interval [100ms] > [11:41:58,631][DEBUG][indices.memory] > [SearchDevelopmentInstance] using index\_buffer\_size [406.2mb], with > min\_shard\_index\_buffer\_size [4mb] > [11:41:58,636][DEBUG][gateway.fs] > [SearchDevelopmentInstance] Latest metadata found at index [-1] > [11:41:58,637][INFO][node] > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: initialized > [11:41:58,637][INFO][node] > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: starting ... > [11:41:58,698][DEBUG][netty.channel.socket.nio.NioProviderMetadata] > Using the autodetected NIO constraint level: 0 > [11:41:58,740][DEBUG][transport.netty] > [SearchDevelopmentInstance] Bound to address [/127.0.0.1:9300] > [11:41:58,741][INFO][transport] > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9300]}, > publish\_address {inet[/127.0.0.1:9300]} > [11:42:01,865][DEBUG][discovery.zen] > [SearchDevelopmentInstance] ping responses: {none} > [11:42:01,870][DEBUG][cluster.service] > [SearchDevelopmentInstance] processing [zen-disco-join > (elected\_as\_master)]: execute > [11:42:01,871][DEBUG][cluster.service] > [SearchDevelopmentInstance] cluster state updated, version [1], source > [zen-disco-join (elected\_as\_master)] > [11:42:01,872][INFO][cluster.service] > [SearchDevelopmentInstance] new\_master [SearchDevelopmentInstance] > [3d1d41e1-4846-4024-8738-230fbd6927b0][inet[/127.0.0.1:9300]], reason: > zen-disco-join (elected\_as\_master) > [11:42:02,014][DEBUG][transport.netty] > [SearchDevelopmentInstance] Connected to node > [[SearchDevelopmentInstance][3d1d41e1-4846-4024-8738-230fbd6927b0] > [inet[/127.0.0.1:9300]]] > [11:42:02,016][DEBUG][cluster.service] > [SearchDevelopmentInstance] processing [zen-disco-join > (elected\_as\_master)]: done applying updated cluster\_state > [11:42:02,016][INFO][discovery] > [SearchDevelopmentInstance] > Ticketfly-1/3d1d41e1-4846-4024-8738-230fbd6927b0 > [11:42:02,016][DEBUG][gateway] > [SearchDevelopmentInstance] reading state from gateway > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... > [11:42:02,017][DEBUG][gateway] > [SearchDevelopmentInstance] read state from gateway > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], took 0s > [11:42:02,017][DEBUG][gateway] > [SearchDevelopmentInstance] no state read from gateway > [11:42:02,018][DEBUG][cluster.service] > [SearchDevelopmentInstance] processing [gateway (marked as read, > reason=no state)]: execute > [11:42:02,019][DEBUG][cluster.service] > [SearchDevelopmentInstance] cluster state updated, version [2], source > [gateway (marked as read, reason=no state)] > [11:42:02,020][DEBUG][gateway] > [SearchDevelopmentInstance] writing to gateway > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... > [11:42:02,020][DEBUG][cluster.service] > [SearchDevelopmentInstance] processing [gateway (marked as read, > reason=no state)]: done applying updated cluster\_state > [11:42:02,023][INFO][http] > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9200]}, > publish\_address {inet[/127.0.0.1:9200]} > [11:42:02,172][DEBUG][gateway] > [SearchDevelopmentInstance] wrote to gateway > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], > took 152ms > [11:42:02,236][INFO][jmx] > [SearchDevelopmentInstance] bound\_address {service:jmx:rmi:///jndi/ > rmi://:9400/jmxrmi}, publish\_address {service:jmx:rmi:///jndi/rmi:// > 127.0.0.1:9400/jmxrmi} > [11:42:02,236][INFO][node] > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: started > [11:42:12,016][DEBUG][cluster.service] > [SearchDevelopmentInstance] processing [routing-table-updater]: > execute > [11:42:12,018][DEBUG][cluster.service] > [SearchDevelopmentInstance] processing [routing-table-updater]: no > change in cluster\_state > > query to get a document I know was there before shutdown > > GIDEON-KAPLANs-MacBook-Pro:~ gideon$ curl -XGET '[http://localhost:9200/](http://localhost:9200/) > backstage/event/11637' > {"error":"IndexMissingException[[backstage] missing]"}GIDEON-KAPLANs- > MacBook-Pro: ---
**Author:** ![Gideon\_Kaplan](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@Gideon\_Kaplan](https://discuss.elastic.co/u/Gideon_Kaplan)\ **Post date:** [October 27, 2010, 7:45pm UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481/3 "2010-10-27T19:45:39Z")
Thanks Shay, I will try the steps you mentioned and update with status (still wrangling my head around the multiple plugins we use here and will need to update some settings / configs to do so) I also need to get a better understanding of the underlying lucene, specifically gateways and stores From the config I posted, having the store as type memory, then a "server" bounce will result in loss of the indexes is the expected behavior? That's the result I see when restarting my entire computer, not just elasticsearch (If I just bounce ES, then the indexes come back) Thanks for the speedy response G On Oct 26, 2:18 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > First, can you try and use the latest version (0.12)? It will be simpler > to support this. > > You have to use the shared gateway (fs) if you are going to store the > index in memory. If not, I recommend starting with the loca gateway (which > is the default out of the box in 0.12). The index local storage (not > gateway) defaults to file system, so you can just remove the store settings > in the index. > > A few points: > > 1. If you are going to store the index in memory, make sure you have enough > of it. > 2. File system index can be pretty fast, make sure you really need the index > to be in memory for the extra juice. > > Also, post shutdown, can you check if the indices are listed in the > metadata file? > > -shay.banon > > On Tue, Oct 26, 2010 at 9:56 PM, Gideon Kaplan [gideonkap...@gmail.com](mailto:gideonkap...@gmail.com)wrote: > > > Hello folks > > I'm running a mac osx 10.6.4 and we are using Elasticsearch > > {elasticsearch/0.9.0}[211]: started > > > We are programmatically building 2 indexes with (sub) types via groovy > > client (and the server has been started separately) > > Everything is hunky dory while the system is running > > I can shut it the terminal window down with ctrl-c or passing the > > shutdown all curl command and restart using elasticsearch -f > > > The problem is when I bounce my system and restart the server it > > indexes are missing > > I believe the gateway is set to FS, but the store is set to memory > > > After bounce I do see some stuff in my work directory > > /usr/local/elasticsearch-0.9.0/work/Ticketfly-1/nodes/0/indices/ > > backstage/1/index > > GIDEON-KAPLANs-MacBook-Pro:index gideon$ ls > > \_0.cfs \_0\_1.del segments.gen segments\_2 > > > But it's looks like it's missing the indices directory in the gateway > > fs store /tmp/elasticsearch/data/cluster > > after system bounce (the below dir is recreated when I rebuild the > > indexes programatically) > > /tmp/elasticsearch/data/cluster/Ticketfly-1/indices > > > This is before rebuilding > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-0 > > { > > "meta-data" : { > > "indices" : { > > } > > } > > }GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ pwd > > /tmp/elasticsearch/data/cluster/Ticketfly-1/metadata > > > This is after > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-5 (metadata-5 > > is the only file there) > > { > > "meta-data" : { > > "indices" : { > > "inventory" : { > > "settings" : { > > "index.number\_of\_shards" : "5", > > "index.number\_of\_replicas" : "4" > > }, > > "mappings" : [ { > > ..... > > > I do see them on the file system though, maybe I'm just doing > > something wrong or is there a way to tell it to rediscover the > > existing indexes on startup or not delete them on shutdown (what I > > think it's doing) > > > Does one have to set the store to fs in the yml file? > > > Sorry if this is a dumb question, I'm a bit new at using this tech > > > my yml file > > name: SearchDevelopmentInstance > > > cluster: > > name: Ticketfly-1 > > > node: > > data: true > > > http: > > enabled: true > > > network: > > #bind\_host: 0.0.0.0 > > #publish\_host: _eth1_ > > host: 127.0.0.1 > > > gateway: > > type: fs > > fs: > > location: /tmp/elasticsearch/data/cluster > > index : > > number\_of\_shards : 5 > > number\_of\_replicas : 4 > > analysis : > > analyzer : > > standard : > > type : standard > > > store: > > type: memory > > memory: > > cache\_size: 100m > > buffer\_size: 10k > > > #transport: > > > > # tcp: > > > > # port: 9300 > > > startup after bounce > > GIDEON-KAPLANs-MacBook-Pro:bin gideon$ elasticsearch -f > > [11:41:56,616][INFO][node] > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: > > initializing ... > > [11:41:56,635][INFO][plugins] > > [SearchDevelopmentInstance] loaded > > [11:41:58,224][DEBUG][cache.memory] > > [SearchDevelopmentInstance] using bytebuffer cache with buffer\_size > > [100kb], cache\_size [200mb], direct [true], warm\_cache [false] > > [11:41:58,272][DEBUG][threadpool.cached] > > [SearchDevelopmentInstance] Initializing cached thread pool with > > keep\_alive[1m], scheduled\_size[20] > > [11:41:58,317][DEBUG][discovery.zen.ping.multicast] > > [SearchDevelopmentInstance] using group [224.2.2.4], with port > > [54328], ttl [3], and address [null] > > [11:41:58,323][DEBUG][discovery.zen.ping.unicast] > > [SearchDevelopmentInstance] using initial hosts > > [11:41:58,333][DEBUG][discovery.zen] > > [SearchDevelopmentInstance] using initial\_ping\_timeout [3s] > > [11:41:58,335][DEBUG][discovery.zen.fd] > > [SearchDevelopmentInstance] [master] uses ping\_interval [1s], > > ping\_timeout [30s], ping\_retries [3] > > [11:41:58,342][DEBUG][discovery.zen.fd] > > [SearchDevelopmentInstance] [node] uses ping\_interval [1s], > > ping\_timeout [30s], ping\_retries [3] > > [11:41:58,410][DEBUG][env] > > [SearchDevelopmentInstance] using node location [/usr/local/ > > elasticsearch-0.9.0/work/Ticketfly-1/nodes/0] > > [11:41:58,504][DEBUG][monitor.memory.alpha] > > [SearchDevelopmentInstance] interval [500ms], upper\_memory\_threshold > > [0.95], lower\_memory\_threshold [0.8], > > translog\_number\_of\_operations\_threshold [5000] > > [11:41:58,536][DEBUG][monitor.network] > > [SearchDevelopmentInstance] net\_info > > host [GIDEON-KAPLANs-MacBook-Pro.local] > > en1 display\_name [en1] > > address [/10.59.15.40] [/ > > fe80:0:0:0:cabc:c8ff:fed9:a56%5] > > mtu [1500] multicast [true] ptp [false] loopback > > [false] up [true] virtual [false] > > en0 display\_name [en0] > > address [/10.56.10.139] [/ > > fe80:0:0:0:cabc:c8ff:fe96:9069%4] > > mtu [1500] multicast [true] ptp [false] loopback > > [false] up [true] virtual [false] > > lo0 display\_name [lo0] > > address [/127.0.0.1] [/fe80:0:0:0:0:0:0:1%1] [/ > > 0:0:0:0:0:0:0:1] > > mtu [16384] multicast [true] ptp [false] loopback > > [true] up [true] virtual [false] > > > [11:41:58,626][DEBUG][indices.recovery.throttler] > > [SearchDevelopmentInstance] concurrent\_recoveries [4], > > concurrent\_streams [4] interval [100ms] > > [11:41:58,631][DEBUG][indices.memory] > > [SearchDevelopmentInstance] using index\_buffer\_size [406.2mb], with > > min\_shard\_index\_buffer\_size [4mb] > > [11:41:58,636][DEBUG][gateway.fs] > > [SearchDevelopmentInstance] Latest metadata found at index [-1] > > [11:41:58,637][INFO][node] > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: initialized > > [11:41:58,637][INFO][node] > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: starting ... > > [11:41:58,698][DEBUG][netty.channel.socket.nio.NioProviderMetadata] > > Using the autodetected NIO constraint level: 0 > > [11:41:58,740][DEBUG][transport.netty] > > [SearchDevelopmentInstance] Bound to address [/127.0.0.1:9300] > > [11:41:58,741][INFO][transport] > > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9300]}, > > publish\_address {inet[/127.0.0.1:9300]} > > [11:42:01,865][DEBUG][discovery.zen] > > [SearchDevelopmentInstance] ping responses: {none} > > [11:42:01,870][DEBUG][cluster.service] > > [SearchDevelopmentInstance] processing [zen-disco-join > > (elected\_as\_master)]: execute > > [11:42:01,871][DEBUG][cluster.service] > > [SearchDevelopmentInstance] cluster state updated, version [1], source > > [zen-disco-join (elected\_as\_master)] > > [11:42:01,872][INFO][cluster.service] > > [SearchDevelopmentInstance] new\_master [SearchDevelopmentInstance] > > [3d1d41e1-4846-4024-8738-230fbd6927b0][inet[/127.0.0.1:9300]], reason: > > zen-disco-join (elected\_as\_master) > > [11:42:02,014][DEBUG][transport.netty] > > [SearchDevelopmentInstance] Connected to node > > [[SearchDevelopmentInstance][3d1d41e1-4846-4024-8738-230fbd6927b0] > > [inet[/127.0.0.1:9300]]] > > [11:42:02,016][DEBUG][cluster.service] > > [SearchDevelopmentInstance] processing [zen-disco-join > > (elected\_as\_master)]: done applying updated cluster\_state > > [11:42:02,016][INFO][discovery] > > [SearchDevelopmentInstance] > > Ticketfly-1/3d1d41e1-4846-4024-8738-230fbd6927b0 > > [11:42:02,016][DEBUG][gateway] > > [SearchDevelopmentInstance] reading state from gateway > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... > > [11:42:02,017][DEBUG][gateway] > > [SearchDevelopmentInstance] read state from gateway > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], took 0s > > [11:42:02,017][DEBUG][gateway] > > [SearchDevelopmentInstance] no state read from gateway > > [11:42:02,018][DEBUG][cluster.service] > > [SearchDevelopmentInstance] processing [gateway (marked as read, > > reason=no state)]: execute > > [11:42:02,019][DEBUG][cluster.service] > > [SearchDevelopmentInstance] cluster state updated, version [2], source > > [gateway (marked as read, reason=no state)] > > [11:42:02,020][DEBUG][gateway] > > [SearchDevelopmentInstance] writing to gateway > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... > > [11:42:02,020][DEBUG][cluster.service] > > [SearchDevelopmentInstance] processing [gateway (marked as read, > > reason=no state)]: done applying updated cluster\_state > > [11:42:02,023][INFO][http] > > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9200]}, > > publish\_address {inet[/127.0.0.1:9200]} > > [11:42:02,172][DEBUG][gateway] > > [SearchDevelopmentInstance] wrote to gateway > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], > > took 152ms > > [11:42:02,236][INFO][jmx] > > [SearchDevelopmentInstance] bound\_address {service:jmx:rmi:///jndi/ > > rmi://:9400/jmxrmi}, publish\_address {service:jmx:rmi:///jndi/rmi:// > > 127.0.0.1:9400/jmxrmi} > > [11:42:02,236][INFO][node] > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: started > > [11:42:12,016][DEBUG][cluster.service] > > [SearchDevelopmentInstance] processing [routing-table-updater]: > > execute > > [11:42:12,018][DEBUG][cluster.service] > > ... > > read more » ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 27, 2010, 8:33pm UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481/4 "2010-10-27T20:33:01Z")
If you have the index store in memory, then you have to have a shared gateway configured (like the fs one). This will survive both ES restart and machine restart (its the same for ES). Assuming you are not deleting the data from the gateway. -shay.banon On Wed, Oct 27, 2010 at 9:45 PM, Gideon Kaplan [gideonkaplan@gmail.com](mailto:gideonkaplan@gmail.com)wrote: > Thanks Shay, > I will try the steps you mentioned and update with status (still > wrangling my head around the multiple plugins we use here and will > need to update some settings / configs to do so) > I also need to get a better understanding of the underlying lucene, > specifically gateways and stores > From the config I posted, having the store as type memory, then a > "server" bounce will result in loss of the indexes is the expected > behavior? > That's the result I see when restarting my entire computer, not just > elasticsearch (If I just bounce ES, then the indexes come back) > Thanks for the speedy response > G > > On Oct 26, 2:18 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hi, > > > > First, can you try and use the latest version (0.12)? It will be > > simpler > > to support this. > > > > You have to use the shared gateway (fs) if you are going to store the > > index in memory. If not, I recommend starting with the loca gateway > > (which > > is the default out of the box in 0.12). The index local storage (not > > gateway) defaults to file system, so you can just remove the store > > settings > > in the index. > > > > A few points: > > > > 1. If you are going to store the index in memory, make sure you have > > enough > > of it. > > 2. File system index can be pretty fast, make sure you really need the > > index > > to be in memory for the extra juice. > > > > Also, post shutdown, can you check if the indices are listed in the > > metadata file? > > > > -shay.banon > > > > On Tue, Oct 26, 2010 at 9:56 PM, Gideon Kaplan \<[gideonkap...@gmail.com](mailto:gideonkap...@gmail.com) > > wrote: > > > > > Hello folks > > > I'm running a mac osx 10.6.4 and we are using Elasticsearch > > > {elasticsearch/0.9.0}[211]: started > > > > > We are programmatically building 2 indexes with (sub) types via groovy > > > client (and the server has been started separately) > > > Everything is hunky dory while the system is running > > > I can shut it the terminal window down with ctrl-c or passing the > > > shutdown all curl command and restart using elasticsearch -f > > > > > The problem is when I bounce my system and restart the server it > > > indexes are missing > > > I believe the gateway is set to FS, but the store is set to memory > > > > > After bounce I do see some stuff in my work directory > > > /usr/local/elasticsearch-0.9.0/work/Ticketfly-1/nodes/0/indices/ > > > backstage/1/index > > > GIDEON-KAPLANs-MacBook-Pro:index gideon$ ls > > > \_0.cfs \_0\_1.del segments.gen segments\_2 > > > > > But it's looks like it's missing the indices directory in the gateway > > > fs store /tmp/elasticsearch/data/cluster > > > after system bounce (the below dir is recreated when I rebuild the > > > indexes programatically) > > > /tmp/elasticsearch/data/cluster/Ticketfly-1/indices > > > > > This is before rebuilding > > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-0 > > > { > > > "meta-data" : { > > > "indices" : { > > > } > > > } > > > }GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ pwd > > > /tmp/elasticsearch/data/cluster/Ticketfly-1/metadata > > > > > This is after > > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-5 (metadata-5 > > > is the only file there) > > > { > > > "meta-data" : { > > > "indices" : { > > > "inventory" : { > > > "settings" : { > > > "index.number\_of\_shards" : "5", > > > "index.number\_of\_replicas" : "4" > > > }, > > > "mappings" : [ { > > > ..... > > > > > I do see them on the file system though, maybe I'm just doing > > > something wrong or is there a way to tell it to rediscover the > > > existing indexes on startup or not delete them on shutdown (what I > > > think it's doing) > > > > > Does one have to set the store to fs in the yml file? > > > > > Sorry if this is a dumb question, I'm a bit new at using this tech > > > > > my yml file > > > name: SearchDevelopmentInstance > > > > > cluster: > > > name: Ticketfly-1 > > > > > node: > > > data: true > > > > > http: > > > enabled: true > > > > > network: > > > #bind\_host: 0.0.0.0 > > > #publish\_host: _eth1_ > > > host: 127.0.0.1 > > > > > gateway: > > > type: fs > > > fs: > > > location: /tmp/elasticsearch/data/cluster > > > index : > > > number\_of\_shards : 5 > > > number\_of\_replicas : 4 > > > analysis : > > > analyzer : > > > standard : > > > type : standard > > > > > store: > > > type: memory > > > memory: > > > cache\_size: 100m > > > buffer\_size: 10k > > > > > #transport: > > > > > > # tcp: > > > > > > # port: 9300 > > > > > startup after bounce > > > GIDEON-KAPLANs-MacBook-Pro:bin gideon$ elasticsearch -f > > > [11:41:56,616][INFO][node] > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: > > > initializing ... > > > [11:41:56,635][INFO][plugins] > > > [SearchDevelopmentInstance] loaded > > > [11:41:58,224][DEBUG][cache.memory] > > > [SearchDevelopmentInstance] using bytebuffer cache with buffer\_size > > > [100kb], cache\_size [200mb], direct [true], warm\_cache [false] > > > [11:41:58,272][DEBUG][threadpool.cached] > > > [SearchDevelopmentInstance] Initializing cached thread pool with > > > keep\_alive[1m], scheduled\_size[20] > > > [11:41:58,317][DEBUG][discovery.zen.ping.multicast] > > > [SearchDevelopmentInstance] using group [224.2.2.4], with port > > > [54328], ttl [3], and address [null] > > > [11:41:58,323][DEBUG][discovery.zen.ping.unicast] > > > [SearchDevelopmentInstance] using initial hosts > > > [11:41:58,333][DEBUG][discovery.zen] > > > [SearchDevelopmentInstance] using initial\_ping\_timeout [3s] > > > [11:41:58,335][DEBUG][discovery.zen.fd] > > > [SearchDevelopmentInstance] [master] uses ping\_interval [1s], > > > ping\_timeout [30s], ping\_retries [3] > > > [11:41:58,342][DEBUG][discovery.zen.fd] > > > [SearchDevelopmentInstance] [node] uses ping\_interval [1s], > > > ping\_timeout [30s], ping\_retries [3] > > > [11:41:58,410][DEBUG][env] > > > [SearchDevelopmentInstance] using node location [/usr/local/ > > > elasticsearch-0.9.0/work/Ticketfly-1/nodes/0] > > > [11:41:58,504][DEBUG][monitor.memory.alpha] > > > [SearchDevelopmentInstance] interval [500ms], upper\_memory\_threshold > > > [0.95], lower\_memory\_threshold [0.8], > > > translog\_number\_of\_operations\_threshold [5000] > > > [11:41:58,536][DEBUG][monitor.network] > > > [SearchDevelopmentInstance] net\_info > > > host [GIDEON-KAPLANs-MacBook-Pro.local] > > > en1 display\_name [en1] > > > address [/10.59.15.40] [/ > > > fe80:0:0:0:cabc:c8ff:fed9:a56%5] > > > mtu [1500] multicast [true] ptp [false] loopback > > > [false] up [true] virtual [false] > > > en0 display\_name [en0] > > > address [/10.56.10.139] [/ > > > fe80:0:0:0:cabc:c8ff:fe96:9069%4] > > > mtu [1500] multicast [true] ptp [false] loopback > > > [false] up [true] virtual [false] > > > lo0 display\_name [lo0] > > > address [/127.0.0.1] [/fe80:0:0:0:0:0:0:1%1] [/ > > > 0:0:0:0:0:0:0:1] > > > mtu [16384] multicast [true] ptp [false] loopback > > > [true] up [true] virtual [false] > > > > > [11:41:58,626][DEBUG][indices.recovery.throttler] > > > [SearchDevelopmentInstance] concurrent\_recoveries [4], > > > concurrent\_streams [4] interval [100ms] > > > [11:41:58,631][DEBUG][indices.memory] > > > [SearchDevelopmentInstance] using index\_buffer\_size [406.2mb], with > > > min\_shard\_index\_buffer\_size [4mb] > > > [11:41:58,636][DEBUG][gateway.fs] > > > [SearchDevelopmentInstance] Latest metadata found at index [-1] > > > [11:41:58,637][INFO][node] > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: initialized > > > [11:41:58,637][INFO][node] > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: starting ... > > > [11:41:58,698][DEBUG][netty.channel.socket.nio.NioProviderMetadata] > > > Using the autodetected NIO constraint level: 0 > > > [11:41:58,740][DEBUG][transport.netty] > > > [SearchDevelopmentInstance] Bound to address [/127.0.0.1:9300] > > > [11:41:58,741][INFO][transport] > > > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9300]}, > > > publish\_address {inet[/127.0.0.1:9300]} > > > [11:42:01,865][DEBUG][discovery.zen] > > > [SearchDevelopmentInstance] ping responses: {none} > > > [11:42:01,870][DEBUG][cluster.service] > > > [SearchDevelopmentInstance] processing [zen-disco-join > > > (elected\_as\_master)]: execute > > > [11:42:01,871][DEBUG][cluster.service] > > > [SearchDevelopmentInstance] cluster state updated, version [1], source > > > [zen-disco-join (elected\_as\_master)] > > > [11:42:01,872][INFO][cluster.service] > > > [SearchDevelopmentInstance] new\_master [SearchDevelopmentInstance] > > > [3d1d41e1-4846-4024-8738-230fbd6927b0][inet[/127.0.0.1:9300]], reason: > > > zen-disco-join (elected\_as\_master) > > > [11:42:02,014][DEBUG][transport.netty] > > > [SearchDevelopmentInstance] Connected to node > > > [[SearchDevelopmentInstance][3d1d41e1-4846-4024-8738-230fbd6927b0] > > > [inet[/127.0.0.1:9300]]] > > > [11:42:02,016][DEBUG][cluster.service] > > > [SearchDevelopmentInstance] processing [zen-disco-join > > > (elected\_as\_master)]: done applying updated cluster\_state > > > [11:42:02,016][INFO][discovery] > > > [SearchDevelopmentInstance] > > > Ticketfly-1/3d1d41e1-4846-4024-8738-230fbd6927b0 > > > [11:42:02,016][DEBUG][gateway] > > > [SearchDevelopmentInstance] reading state from gateway > > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... > > > [11:42:02,017][DEBUG][gateway] > > > [SearchDevelopmentInstance] read state from gateway > > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], took 0s > > > [11:42:02,017][DEBUG][gateway] > > > [SearchDevelopmentInstance] no state read from gateway > > > [11:42:02,018][DEBUG][cluster.service] > > > [SearchDevelopmentInstance] processing [gateway (marked as read, > > > reason=no state)]: execute > > > [11:42:02,019][DEBUG][cluster.service] > > > [SearchDevelopmentInstance] cluster state updated, version [2], source > > > [gateway (marked as read, reason=no state)] > > > [11:42:02,020][DEBUG][gateway] > > > [SearchDevelopmentInstance] writing to gateway > > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1] ... > > > [11:42:02,020][DEBUG][cluster.service] > > > [SearchDevelopmentInstance] processing [gateway (marked as read, > > > reason=no state)]: done applying updated cluster\_state > > > [11:42:02,023][INFO][http] > > > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9200]}, > > > publish\_address {inet[/127.0.0.1:9200]} > > > [11:42:02,172][DEBUG][gateway] > > > [SearchDevelopmentInstance] wrote to gateway > > > fs:///tmp/elasticsearch/data/cluster/[Ticketfly-1], > > > took 152ms > > > [11:42:02,236][INFO][jmx] > > > [SearchDevelopmentInstance] bound\_address {service:jmx:rmi:///jndi/ > > > rmi://:9400/jmxrmi}, publish\_address {service:jmx:rmi:///jndi/rmi:// > > > 127.0.0.1:9400/jmxrmi} > > > [11:42:02,236][INFO][node] > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: started > > > [11:42:12,016][DEBUG][cluster.service] > > > [SearchDevelopmentInstance] processing [routing-table-updater]: > > > execute > > > [11:42:12,018][DEBUG][cluster.service] > > > > ... > > > > read more » ---
**Author:** ![Gideon\_Kaplan](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@Gideon\_Kaplan](https://discuss.elastic.co/u/Gideon_Kaplan)\ **Post date:** [October 27, 2010, 9:51pm UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481/5 "2010-10-27T21:51:20Z")
hmm, That's not the behavior Im seeing when doing a computer restart, It's only durable if I bounce ES, but leave the computer up, maybe because its configured to us a sub dir or /tmp? But I do see the other files persists Will try and update the plugin first with the same config , ie a memory-based store and see if that fixes it Maybe theres also some functionality the deletes the indeces on shutdown that I am not aware of On Oct 27, 1:33 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > If you have the index store in memory, then you have to have a shared > gateway configured (like the fs one). This will survive both ES restart and > machine restart (its the same for ES). Assuming you are not deleting the > data from the gateway. > > -shay.banon > > On Wed, Oct 27, 2010 at 9:45 PM, Gideon Kaplan [gideonkap...@gmail.com](mailto:gideonkap...@gmail.com)wrote: > > > Thanks Shay, > > I will try the steps you mentioned and update with status (still > > wrangling my head around the multiple plugins we use here and will > > need to update some settings / configs to do so) > > I also need to get a better understanding of the underlying lucene, > > specifically gateways and stores > > From the config I posted, having the store as type memory, then a > > "server" bounce will result in loss of the indexes is the expected > > behavior? > > That's the result I see when restarting my entire computer, not just > > elasticsearch (If I just bounce ES, then the indexes come back) > > Thanks for the speedy response > > G > > > On Oct 26, 2:18 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Hi, > > > > First, can you try and use the latest version (0.12)? It will be > > > simpler > > > to support this. > > > > You have to use the shared gateway (fs) if you are going to store the > > > index in memory. If not, I recommend starting with the loca gateway > > > (which > > > is the default out of the box in 0.12). The index local storage (not > > > gateway) defaults to file system, so you can just remove the store > > > settings > > > in the index. > > > > A few points: > > > > 1. If you are going to store the index in memory, make sure you have > > > enough > > > of it. > > > 2. File system index can be pretty fast, make sure you really need the > > > index > > > to be in memory for the extra juice. > > > > Also, post shutdown, can you check if the indices are listed in the > > > metadata file? > > > > -shay.banon > > > > On Tue, Oct 26, 2010 at 9:56 PM, Gideon Kaplan \<[gideonkap...@gmail.com](mailto:gideonkap...@gmail.com) > > > wrote: > > > > > Hello folks > > > > I'm running a mac osx 10.6.4 and we are using Elasticsearch > > > > {elasticsearch/0.9.0}[211]: started > > > > > We are programmatically building 2 indexes with (sub) types via groovy > > > > client (and the server has been started separately) > > > > Everything is hunky dory while the system is running > > > > I can shut it the terminal window down with ctrl-c or passing the > > > > shutdown all curl command and restart using elasticsearch -f > > > > > The problem is when I bounce my system and restart the server it > > > > indexes are missing > > > > I believe the gateway is set to FS, but the store is set to memory > > > > > After bounce I do see some stuff in my work directory > > > > /usr/local/elasticsearch-0.9.0/work/Ticketfly-1/nodes/0/indices/ > > > > backstage/1/index > > > > GIDEON-KAPLANs-MacBook-Pro:index gideon$ ls > > > > \_0.cfs \_0\_1.del segments.gen segments\_2 > > > > > But it's looks like it's missing the indices directory in the gateway > > > > fs store /tmp/elasticsearch/data/cluster > > > > after system bounce (the below dir is recreated when I rebuild the > > > > indexes programatically) > > > > /tmp/elasticsearch/data/cluster/Ticketfly-1/indices > > > > > This is before rebuilding > > > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-0 > > > > { > > > > "meta-data" : { > > > > "indices" : { > > > > } > > > > } > > > > }GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ pwd > > > > /tmp/elasticsearch/data/cluster/Ticketfly-1/metadata > > > > > This is after > > > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-5 (metadata-5 > > > > is the only file there) > > > > { > > > > "meta-data" : { > > > > "indices" : { > > > > "inventory" : { > > > > "settings" : { > > > > "index.number\_of\_shards" : "5", > > > > "index.number\_of\_replicas" : "4" > > > > }, > > > > "mappings" : [ { > > > > ..... > > > > > I do see them on the file system though, maybe I'm just doing > > > > something wrong or is there a way to tell it to rediscover the > > > > existing indexes on startup or not delete them on shutdown (what I > > > > think it's doing) > > > > > Does one have to set the store to fs in the yml file? > > > > > Sorry if this is a dumb question, I'm a bit new at using this tech > > > > > my yml file > > > > name: SearchDevelopmentInstance > > > > > cluster: > > > > name: Ticketfly-1 > > > > > node: > > > > data: true > > > > > http: > > > > enabled: true > > > > > network: > > > > #bind\_host: 0.0.0.0 > > > > #publish\_host: _eth1_ > > > > host: 127.0.0.1 > > > > > gateway: > > > > type: fs > > > > fs: > > > > location: /tmp/elasticsearch/data/cluster > > > > index : > > > > number\_of\_shards : 5 > > > > number\_of\_replicas : 4 > > > > analysis : > > > > analyzer : > > > > standard : > > > > type : standard > > > > > store: > > > > type: memory > > > > memory: > > > > cache\_size: 100m > > > > buffer\_size: 10k > > > > > #transport: > > > > > > > > # tcp: > > > > > > > > # port: 9300 > > > > > startup after bounce > > > > GIDEON-KAPLANs-MacBook-Pro:bin gideon$ elasticsearch -f > > > > [11:41:56,616][INFO][node] > > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: > > > > initializing ... > > > > [11:41:56,635][INFO][plugins] > > > > [SearchDevelopmentInstance] loaded > > > > [11:41:58,224][DEBUG][cache.memory] > > > > [SearchDevelopmentInstance] using bytebuffer cache with buffer\_size > > > > [100kb], cache\_size [200mb], direct [true], warm\_cache [false] > > > > [11:41:58,272][DEBUG][threadpool.cached] > > > > [SearchDevelopmentInstance] Initializing cached thread pool with > > > > keep\_alive[1m], scheduled\_size[20] > > > > [11:41:58,317][DEBUG][discovery.zen.ping.multicast] > > > > [SearchDevelopmentInstance] using group [224.2.2.4], with port > > > > [54328], ttl [3], and address [null] > > > > [11:41:58,323][DEBUG][discovery.zen.ping.unicast] > > > > [SearchDevelopmentInstance] using initial hosts > > > > [11:41:58,333][DEBUG][discovery.zen] > > > > [SearchDevelopmentInstance] using initial\_ping\_timeout [3s] > > > > [11:41:58,335][DEBUG][discovery.zen.fd] > > > > [SearchDevelopmentInstance] [master] uses ping\_interval [1s], > > > > ping\_timeout [30s], ping\_retries [3] > > > > [11:41:58,342][DEBUG][discovery.zen.fd] > > > > [SearchDevelopmentInstance] [node] uses ping\_interval [1s], > > > > ping\_timeout [30s], ping\_retries [3] > > > > [11:41:58,410][DEBUG][env] > > > > [SearchDevelopmentInstance] using node location [/usr/local/ > > > > elasticsearch-0.9.0/work/Ticketfly-1/nodes/0] > > > > [11:41:58,504][DEBUG][monitor.memory.alpha] > > > > [SearchDevelopmentInstance] interval [500ms], upper\_memory\_threshold > > > > [0.95], lower\_memory\_threshold [0.8], > > > > translog\_number\_of\_operations\_threshold [5000] > > > > [11:41:58,536][DEBUG][monitor.network] > > > > [SearchDevelopmentInstance] net\_info > > > > host [GIDEON-KAPLANs-MacBook-Pro.local] > > > > en1 display\_name [en1] > > > > address [/10.59.15.40] [/ > > > > fe80:0:0:0:cabc:c8ff:fed9:a56%5] > > > > mtu [1500] multicast [true] ptp [false] loopback > > > > [false] up [true] virtual [false] > > > > en0 display\_name [en0] > > > > address [/10.56.10.139] [/ > > > > fe80:0:0:0:cabc:c8ff:fe96:9069%4] > > > > mtu [1500] multicast [true] ptp [false] loopback > > > > [false] up [true] virtual [false] > > > > lo0 display\_name [lo0] > > > > address [/127.0.0.1] [/fe80:0:0:0:0:0:0:1%1] [/ > > > > 0:0:0:0:0:0:0:1] > > > > mtu [16384] multicast [true] ptp [false] loopback > > > > [true] up [true] virtual [false] > > > > > [11:41:58,626][DEBUG][indices.recovery.throttler] > > > > [SearchDevelopmentInstance] concurrent\_recoveries [4], > > > > concurrent\_streams [4] interval [100ms] > > > > [11:41:58,631][DEBUG][indices.memory] > > > > [SearchDevelopmentInstance] using index\_buffer\_size [406.2mb], with > > > > min\_shard\_index\_buffer\_size [4mb] > > > > [11:41:58,636][DEBUG][gateway.fs] > > > > [SearchDevelopmentInstance] Latest metadata found at index [-1] > > > > [11:41:58,637][INFO][node] > > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: initialized > > > > [11:41:58,637][INFO][node] > > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: starting ... > > > > [11:41:58,698][DEBUG][netty.channel.socket.nio.NioProviderMetadata] > > > > Using the autodetected NIO constraint level: 0 > > > > [11:41:58,740][DEBUG][transport.netty] > > > > [SearchDevelopmentInstance] Bound to address [/127.0.0.1:9300] > > > > [11:41:58,741][INFO][transport] > > > > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9300]}, > > > > publish\_address {inet[/127.0.0.1:9300]} > > > > [11:42:01,865][DEBUG][discovery.zen] > > > > [SearchDevelopmentInstance] ping responses: {none} > > > > [11:42:01,870][DEBUG][cluster.service] > > > > [SearchDevelopmentInstance] processing [zen-disco-join > > > > (elected\_as\_master)]: execute > > > > [11:42:01,871][DEBUG][cluster.service] > > > > [SearchDevelopmentInstance] cluster state updated, version [1], source > > > > [zen-disco-join (elected\_as\_master)] > > > > [11:42:01,872][INFO][cluster.service] > > > > [SearchDevelopmentInstance] new\_master [SearchDevelopmentInstance] > > > > [3d1d41e1-4846-4024-8738-230fbd6927b0][inet[/127.0.0.1:9300]], reason: > > > > zen-disco-join (elected\_as\_master) > > > > [11:42:02,014][DEBUG][transport.netty] > > > > [SearchDevelopmentInstance] Connected to node > > > > [[SearchDevelopmentInstance][3d1d41e1-4846-4024-8738-230fbd6927b0] > > > > [inet[/127.0.0.1:9300]]] > > > > [11:42:02,016][DEBUG][cluster.service] > > > > [SearchDevelopmentInstance] processing [zen-disco-join > > > > (elected\_as\_master)]: done applying updated cluster\_state > > > > [11:42:02,016][INFO][discovery] > > > > [SearchDevelopmentInstance] > > > > Ticketfly-1/3d1d41e1-4846-4024-8738-230fbd6927b0 > > > > [11:42:02,016][DEBUG][gateway > > ... > > read more » ---
**Author:** ![Gideon\_Kaplan](https://avatars.discourse-cdn.com/v4/letter/g/779978/32.png) [@Gideon\_Kaplan](https://discuss.elastic.co/u/Gideon_Kaplan)\ **Post date:** [October 28, 2010, 6:45pm UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481/6 "2010-10-28T18:45:12Z")
Hi Again Think I found out why the indices were lost, was setting the gateway file location to a subdir of /tmp on mac, which apparently gets deleted on some schedule changing the yml config to seem to allow it survive a computer restart old config gateway: type: fs fs: location: /tmp/elasticsearch/data/cluster new config gateway: type: fs fs: location: /usr/local/elasticsearch-0.9.0/data/cluster This is one article I found about deleting /tmp on mac os x [http://efreedom.com/Question/3-187071/Mac-OS-106-Often-Tmp-Deleted](http://efreedom.com/Question/3-187071/Mac-OS-106-Often-Tmp-Deleted) On Oct 27, 2:51 pm, Gideon Kaplan [gideonkap...@gmail.com](mailto:gideonkap...@gmail.com) wrote: > hmm, > That's not the behavior Im seeing when doing a computer restart, It's > only durable if I bounce ES, but leave the computer up, maybe because > its configured to us a sub dir or /tmp? But I do see the other files > persists > Will try and update the plugin first with the same config , ie a > memory-based store and see if that fixes it > Maybe theres also some functionality the deletes the indeces on > shutdown that I am not aware of > > On Oct 27, 1:33 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > If you have the index store in memory, then you have to have a shared > > gateway configured (like the fs one). This will survive both ES restart and > > machine restart (its the same for ES). Assuming you are not deleting the > > data from the gateway. > > > -shay.banon > > > On Wed, Oct 27, 2010 at 9:45 PM, Gideon Kaplan [gideonkap...@gmail.com](mailto:gideonkap...@gmail.com)wrote: > > > > Thanks Shay, > > > I will try the steps you mentioned and update with status (still > > > wrangling my head around the multiple plugins we use here and will > > > need to update some settings / configs to do so) > > > I also need to get a better understanding of the underlying lucene, > > > specifically gateways and stores > > > From the config I posted, having the store as type memory, then a > > > "server" bounce will result in loss of the indexes is the expected > > > behavior? > > > That's the result I see when restarting my entire computer, not just > > > elasticsearch (If I just bounce ES, then the indexes come back) > > > Thanks for the speedy response > > > G > > > > On Oct 26, 2:18 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Hi, > > > > > First, can you try and use the latest version (0.12)? It will be > > > > simpler > > > > to support this. > > > > > You have to use the shared gateway (fs) if you are going to store the > > > > index in memory. If not, I recommend starting with the loca gateway > > > > (which > > > > is the default out of the box in 0.12). The index local storage (not > > > > gateway) defaults to file system, so you can just remove the store > > > > settings > > > > in the index. > > > > > A few points: > > > > > 1. If you are going to store the index in memory, make sure you have > > > > enough > > > > of it. > > > > 2. File system index can be pretty fast, make sure you really need the > > > > index > > > > to be in memory for the extra juice. > > > > > Also, post shutdown, can you check if the indices are listed in the > > > > metadata file? > > > > > -shay.banon > > > > > On Tue, Oct 26, 2010 at 9:56 PM, Gideon Kaplan \<[gideonkap...@gmail.com](mailto:gideonkap...@gmail.com) > > > > wrote: > > > > > > Hello folks > > > > > I'm running a mac osx 10.6.4 and we are using Elasticsearch > > > > > {elasticsearch/0.9.0}[211]: started > > > > > > We are programmatically building 2 indexes with (sub) types via groovy > > > > > client (and the server has been started separately) > > > > > Everything is hunky dory while the system is running > > > > > I can shut it the terminal window down with ctrl-c or passing the > > > > > shutdown all curl command and restart using elasticsearch -f > > > > > > The problem is when I bounce my system and restart the server it > > > > > indexes are missing > > > > > I believe the gateway is set to FS, but the store is set to memory > > > > > > After bounce I do see some stuff in my work directory > > > > > /usr/local/elasticsearch-0.9.0/work/Ticketfly-1/nodes/0/indices/ > > > > > backstage/1/index > > > > > GIDEON-KAPLANs-MacBook-Pro:index gideon$ ls > > > > > \_0.cfs \_0\_1.del segments.gen segments\_2 > > > > > > But it's looks like it's missing the indices directory in the gateway > > > > > fs store /tmp/elasticsearch/data/cluster > > > > > after system bounce (the below dir is recreated when I rebuild the > > > > > indexes programatically) > > > > > /tmp/elasticsearch/data/cluster/Ticketfly-1/indices > > > > > > This is before rebuilding > > > > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-0 > > > > > { > > > > > "meta-data" : { > > > > > "indices" : { > > > > > } > > > > > } > > > > > }GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ pwd > > > > > /tmp/elasticsearch/data/cluster/Ticketfly-1/metadata > > > > > > This is after > > > > > GIDEON-KAPLANs-MacBook-Pro:metadata gideon$ cat metadata-5 (metadata-5 > > > > > is the only file there) > > > > > { > > > > > "meta-data" : { > > > > > "indices" : { > > > > > "inventory" : { > > > > > "settings" : { > > > > > "index.number\_of\_shards" : "5", > > > > > "index.number\_of\_replicas" : "4" > > > > > }, > > > > > "mappings" : [ { > > > > > ..... > > > > > > I do see them on the file system though, maybe I'm just doing > > > > > something wrong or is there a way to tell it to rediscover the > > > > > existing indexes on startup or not delete them on shutdown (what I > > > > > think it's doing) > > > > > > Does one have to set the store to fs in the yml file? > > > > > > Sorry if this is a dumb question, I'm a bit new at using this tech > > > > > > my yml file > > > > > name: SearchDevelopmentInstance > > > > > > cluster: > > > > > name: Ticketfly-1 > > > > > > node: > > > > > data: true > > > > > > http: > > > > > enabled: true > > > > > > network: > > > > > #bind\_host: 0.0.0.0 > > > > > #publish\_host: _eth1_ > > > > > host: 127.0.0.1 > > > > > > gateway: > > > > > type: fs > > > > > fs: > > > > > location: /tmp/elasticsearch/data/cluster > > > > > index : > > > > > number\_of\_shards : 5 > > > > > number\_of\_replicas : 4 > > > > > analysis : > > > > > analyzer : > > > > > standard : > > > > > type : standard > > > > > > store: > > > > > type: memory > > > > > memory: > > > > > cache\_size: 100m > > > > > buffer\_size: 10k > > > > > > #transport: > > > > > > > > > > # tcp: > > > > > > > > > > # port: 9300 > > > > > > startup after bounce > > > > > GIDEON-KAPLANs-MacBook-Pro:bin gideon$ elasticsearch -f > > > > > [11:41:56,616][INFO][node] > > > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: > > > > > initializing ... > > > > > [11:41:56,635][INFO][plugins] > > > > > [SearchDevelopmentInstance] loaded > > > > > [11:41:58,224][DEBUG][cache.memory] > > > > > [SearchDevelopmentInstance] using bytebuffer cache with buffer\_size > > > > > [100kb], cache\_size [200mb], direct [true], warm\_cache [false] > > > > > [11:41:58,272][DEBUG][threadpool.cached] > > > > > [SearchDevelopmentInstance] Initializing cached thread pool with > > > > > keep\_alive[1m], scheduled\_size[20] > > > > > [11:41:58,317][DEBUG][discovery.zen.ping.multicast] > > > > > [SearchDevelopmentInstance] using group [224.2.2.4], with port > > > > > [54328], ttl [3], and address [null] > > > > > [11:41:58,323][DEBUG][discovery.zen.ping.unicast] > > > > > [SearchDevelopmentInstance] using initial hosts > > > > > [11:41:58,333][DEBUG][discovery.zen] > > > > > [SearchDevelopmentInstance] using initial\_ping\_timeout [3s] > > > > > [11:41:58,335][DEBUG][discovery.zen.fd] > > > > > [SearchDevelopmentInstance] [master] uses ping\_interval [1s], > > > > > ping\_timeout [30s], ping\_retries [3] > > > > > [11:41:58,342][DEBUG][discovery.zen.fd] > > > > > [SearchDevelopmentInstance] [node] uses ping\_interval [1s], > > > > > ping\_timeout [30s], ping\_retries [3] > > > > > [11:41:58,410][DEBUG][env] > > > > > [SearchDevelopmentInstance] using node location [/usr/local/ > > > > > elasticsearch-0.9.0/work/Ticketfly-1/nodes/0] > > > > > [11:41:58,504][DEBUG][monitor.memory.alpha] > > > > > [SearchDevelopmentInstance] interval [500ms], upper\_memory\_threshold > > > > > [0.95], lower\_memory\_threshold [0.8], > > > > > translog\_number\_of\_operations\_threshold [5000] > > > > > [11:41:58,536][DEBUG][monitor.network] > > > > > [SearchDevelopmentInstance] net\_info > > > > > host [GIDEON-KAPLANs-MacBook-Pro.local] > > > > > en1 display\_name [en1] > > > > > address [/10.59.15.40] [/ > > > > > fe80:0:0:0:cabc:c8ff:fed9:a56%5] > > > > > mtu [1500] multicast [true] ptp [false] loopback > > > > > [false] up [true] virtual [false] > > > > > en0 display\_name [en0] > > > > > address [/10.56.10.139] [/ > > > > > fe80:0:0:0:cabc:c8ff:fe96:9069%4] > > > > > mtu [1500] multicast [true] ptp [false] loopback > > > > > [false] up [true] virtual [false] > > > > > lo0 display\_name [lo0] > > > > > address [/127.0.0.1] [/fe80:0:0:0:0:0:0:1%1] [/ > > > > > 0:0:0:0:0:0:0:1] > > > > > mtu [16384] multicast [true] ptp [false] loopback > > > > > [true] up [true] virtual [false] > > > > > > [11:41:58,626][DEBUG][indices.recovery.throttler] > > > > > [SearchDevelopmentInstance] concurrent\_recoveries [4], > > > > > concurrent\_streams [4] interval [100ms] > > > > > [11:41:58,631][DEBUG][indices.memory] > > > > > [SearchDevelopmentInstance] using index\_buffer\_size [406.2mb], with > > > > > min\_shard\_index\_buffer\_size [4mb] > > > > > [11:41:58,636][DEBUG][gateway.fs] > > > > > [SearchDevelopmentInstance] Latest metadata found at index [-1] > > > > > [11:41:58,637][INFO][node] > > > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: initialized > > > > > [11:41:58,637][INFO][node] > > > > > [SearchDevelopmentInstance] {elasticsearch/0.9.0}[211]: starting ... > > > > > [11:41:58,698][DEBUG][netty.channel.socket.nio.NioProviderMetadata] > > > > > Using the autodetected NIO constraint level: 0 > > > > > [11:41:58,740][DEBUG][transport.netty] > > > > > [SearchDevelopmentInstance] Bound to address [/127.0.0.1:9300] > > > > > [11:41:58,741][INFO][transport] > > > > > [SearchDevelopmentInstance] bound\_address {inet[/127.0.0.1:9300]}, > > ... > > read more » ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:17am UTC](https://discuss.elastic.co/t/indices-lost-after-system-bounce-missingindexexception/3481/7 "2017-07-06T04:17:28Z")