# ES snapshots **URL:** \ **Category:** Elasticsearch\ **Created:** [May 26, 2015, 6:12am UTC](https://discuss.elastic.co/t/es-snapshots/1281 "2015-05-26T06:12:13Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![it2](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@it2](https://discuss.elastic.co/u/it2)\ **Post date:** [May 26, 2015, 6:12am UTC](https://discuss.elastic.co/t/es-snapshots/1281/1 "2015-05-26T06:12:13Z")
Hello there! I have cluster with ES 1.3.8, kibana3 and Logstash 1.5.0 on board. I also have a remote space for snapshots. It's mounted by sshfs for whole cluster. but when I starting up a snapshot, my cluster is stopping. Logs are not collecting. And only after snapshot and when I restart services, everything back in normal. So when snapshont is making (~2 hours) - I havenot logs for this time interval and this is sad ☹ Please, any advice. thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 26, 2015, 7:25am UTC](https://discuss.elastic.co/t/es-snapshots/1281/2 "2015-05-26T07:25:40Z")
Are any of the APIs responding, \_cat/nodes, hot\_threads etc etc? ---
**Author:** ![it2](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@it2](https://discuss.elastic.co/u/it2)\ **Post date:** [May 26, 2015, 10:53am UTC](https://discuss.elastic.co/t/es-snapshots/1281/3 "2015-05-26T10:53:40Z")
When I will make next snapshot (soon), I will see. THanks! ---
**Author:** ![it2](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@it2](https://discuss.elastic.co/u/it2)\ **Post date:** [May 26, 2015, 11:21am UTC](https://discuss.elastic.co/t/es-snapshots/1281/4 "2015-05-26T11:21:36Z")
I started a snapshot and everything broked. But cat says everything is ok curl localhost:9200/\_cat/nodes?h=n,ip,port,v,d,hp,m data2 192.168.98.8 9300 1.3.2 398.6gb 71 - data1 192.168.98.7 9300 1.3.2 385gb 42 - logstash 127.0.0.1 9300 1.3.8 17.7gb 4 \* logstash-localhost.localdomain-2447-7946 127.0.0.1 9301 1.5.1 -1b 33 - ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 26, 2015, 11:44pm UTC](https://discuss.elastic.co/t/es-snapshots/1281/5 "2015-05-26T23:44:12Z")
What state does your cluster go into? Are you monitoring load on the nodes? ---
**Author:** ![it2](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@it2](https://discuss.elastic.co/u/it2)\ **Post date:** [May 27, 2015, 6:59am UTC](https://discuss.elastic.co/t/es-snapshots/1281/6 "2015-05-27T06:59:58Z")
I monitor my cluster by kopf plugin. When snapshot begin, nothing changes. Everything is green, and i havenot load on my nodes. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 27, 2015, 10:13am UTC](https://discuss.elastic.co/t/es-snapshots/1281/7 "2015-05-27T10:13:22Z")
> When snapshot begin, nothing changes. Everything is green, and i havenot load on my nodes. What if you look at the [snapshot status](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_snapshot_status)? Snapshotting doesn't necessarily place a heavy load on the nodes, and it definitely shouldn't affect the cluster's status (if that's what you were referring to with "green"). ---
**Author:** ![it2](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@it2](https://discuss.elastic.co/u/it2)\ **Post date:** [May 27, 2015, 1:59pm UTC](https://discuss.elastic.co/t/es-snapshots/1281/8 "2015-05-27T13:59:38Z")
curl -XGET "localhost:9200/\_snapshot/4tbtest/15.03.20-22/\_status" ``` {"snapshots":[{"snapshot":"15.03.20-22","repository":"4tbtest","state":"STARTED","shards_stats":{"initializing":4,"started":8,"finalizing":0,"done":0,"failed":0,"total":12},"stats":{"number_of_files":2326,"processed_files":185,"total_size_in_bytes":77395626630,"processed_size_in_bytes":394038158,"start_time_in_millis":1432734868342,"time_in_millis":0},"indices":{"logstash-2015.03.20":{"shards_stats":{"initializing":1,"started":3,"finalizing":0,"done":0,"failed":0,"total":4},"stats":{"number_of_files":881,"processed_files":76,"total_size_in_bytes":30777452788,"processed_size_in_bytes":92625050,"start_time_in_millis":1432734868088,"time_in_millis":257},"shards":{"0":{"stage":"INIT","stats":{"number_of_files":0,"processed_files":0,"total_size_in_bytes":0,"processed_size_in_bytes":0,"start_time_in_millis":0,"time_in_millis":0},"node":"v7d4Ya0jTcWzFObkh9UDJA"},"1":{"stage":"STARTED","stats":{"number_of_files":211,"processed_files":76,"total_size_in_bytes":10237498758,"processed_size_in_bytes":92625050,"start_time_in_millis":1432734868088,"time_in_millis":0},"node":"XNs4qFu2SNK126PyfAOZdg"},"2":{"stage":"STARTED","stats":{"number_of_files":329,"processed_files":0,"total_size_in_bytes":10268019904,"processed_size_in_bytes":0,"start_time_in_millis":1432734868345,"time_in_millis":0},"node":"v7d4Ya0jTcWzFObkh9UDJA"},"3":{"stage":"STARTED","stats":{"number_of_files":341,"processed_files":0,"total_size_in_bytes":10271934126,"processed_size_in_bytes":0,"start_time_in_millis":1432734868091,"time_in_millis":0},"node":"XNs4qFu2SNK126PyfAOZdg"}}},"logstash-2015.03.22":{"shards_stats":{"initializing":1,"started":3,"finalizing":0,"done":0,"failed":0,"total":4},"stats":{"number_of_files":775,"processed_files":109,"total_size_in_bytes":26597219323,"processed_size_in_bytes":301413108,"start_time_in_millis":1432734868342,"time_in_millis":0},"shards":{"0":{"stage":"STARTED","stats":{"number_of_files":328,"processed_files":0,"total_size_in_bytes":8877532129,"processed_size_in_bytes":0,"start_time_in_millis":1432734868094,"time_in_millis":0},"node":"XNs4qFu2SNK126PyfAOZdg"},"1":{"stage":"STARTED","stats":{"number_of_files":230,"processed_files":0,"total_size_in_bytes":8861607395,"processed_size_in_bytes":0,"start_time_in_millis":1432734868347,"time_in_millis":0},"node":"v7d4Ya0jTcWzFObkh9UDJA"},"2":{"stage":"INIT","stats":{"number_of_files":0,"processed_files":0,"total_size_in_bytes":0,"processed_size_in_bytes":0,"start_time_in_millis":0,"time_in_millis":0},"node":"XNs4qFu2SNK126PyfAOZdg"},"3":{"stage":"STARTED","stats":{"number_of_files":217,"processed_files":109,"total_size_in_bytes":8858079799,"processed_size_in_bytes":301413108,"start_time_in_millis":1432734868342,"time_in_millis":0},"node":"v7d4Ya0jTcWzFObkh9UDJA"}}},"logstash-2015.03.21":{"shards_stats":{"initializing":2,"started":2,"finalizing":0,"done":0,"failed":0,"total":4},"stats":{"number_of_files":670,"processed_files":0,"total_size_in_bytes":20020954519,"processed_size_in_bytes":0,"start_time_in_millis":1432734868098,"time_in_millis":246},"shards":{"0":{"stage":"INIT","stats":{"number_of_files":0,"processed_files":0,"total_size_in_bytes":0,"processed_size_in_bytes":0,"start_time_in_millis":0,"time_in_millis":0},"node":"XNs4qFu2SNK126PyfAOZdg"},"1":{"stage":"INIT","stats":{"number_of_files":0,"processed_files":0,"total_size_in_bytes":0,"processed_size_in_bytes":0,"start_time_in_millis":0,"time_in_millis":0},"node":"v7d4Ya0jTcWzFObkh9UDJA"},"2":{"stage":"STARTED","stats":{"number_of_files":332,"processed_files":0,"total_size_in_bytes":10000314469,"processed_size_in_bytes":0,"start_time_in_millis":1432734868098,"time_in_millis":0},"node":"XNs4qFu2SNK126PyfAOZdg"},"3":{"stage":"STARTED","stats":{"number_of_files":338,"processed_files":0,"total_size_in_bytes":10020640050,"processed_size_in_bytes":0,"start_time_in_millis":1432734868344,"time_in_millis":0},"node":"v7d4Ya0jTcWzFObkh9UDJA"}}}}}]} ``` and now my cluster looks like: [http://prntscr.com/79zoa1](http://prntscr.com/79zoa1) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 27, 2015, 8:03pm UTC](https://discuss.elastic.co/t/es-snapshots/1281/9 "2015-05-27T20:03:22Z")
> curl -XGET "localhost:9200/\_snapshot/4tbtest/15.03.20-22/\_status" That looks perfectly normal. > and now my cluster looks like: [Screenshot by Lightshot](http://prntscr.com/79zoa1) And you're saying that the drop in messages correlate with the start of the snapshot activity...? ---
**Author:** ![it2](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@it2](https://discuss.elastic.co/u/it2)\ **Post date:** [May 28, 2015, 6:36am UTC](https://discuss.elastic.co/t/es-snapshots/1281/10 "2015-05-28T06:36:49Z")
> And you're saying that the drop in messages correlate with the start of the snapshot activity...? Yeap, you right. I cant associate it with something else. Maybe something with my config? ``` cluster.name: elasticsearch2 node.name: "logstash" script.disable_dynamic: true #This forces the JVM to allocate all of ES_MIN_MEM immediately. bootstrap.mlockall: true node.master: true node.data: false index.number_of_shards: 4 index.number_of_replicas: 0 index.refresh_interval: 30s # Search pool threadpool.search.type: fixed threadpool.search.size: 20 threadpool.search.queue_size: 100 # Bulk pool threadpool.bulk.type: fixed threadpool.bulk.size: 60 threadpool.bulk.queue_size: 300 # Index pool threadpool.index.type: fixed threadpool.index.size: 20 threadpool.index.queue_size: 100 # Indices settings indices.memory.index_buffer_size: 30% indices.memory.min_shard_index_buffer_size: 12mb indices.memory.min_index_buffer_size: 96mb # Cache Sizes indices.fielddata.cache.size: 15% indices.fielddata.cache.expire: 6h indices.cache.filter.size: 15% indices.cache.filter.expire: 6h # Indexing Settings for Writes index.refresh_interval: 30s index.translog.flush_threshold_ops: 50000 index.search.slowlog.threshold.query.warn: 10s index.search.slowlog.threshold.query.info: 5s index.search.slowlog.threshold.query.debug: 2s index.search.slowlog.threshold.query.trace: 500ms index.search.slowlog.threshold.fetch.warn: 1s index.search.slowlog.threshold.fetch.info: 800ms index.search.slowlog.threshold.fetch.debug: 500ms index.search.slowlog.threshold.fetch.trace: 200ms index.indexing.slowlog.threshold.index.warn: 10s index.indexing.slowlog.threshold.index.info: 5s index.indexing.slowlog.threshold.index.debug: 2s index.indexing.slowlog.threshold.index.trace: 500ms http.cors.allow-origin: "*" http.cors.enabled: true ``` ---
**Author:** ![it2](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@it2](https://discuss.elastic.co/u/it2)\ **Post date:** [May 28, 2015, 6:46am UTC](https://discuss.elastic.co/t/es-snapshots/1281/11 "2015-05-28T06:46:44Z")
I also want to describe the whole process. I'm snapshoting my old indexes which are closed. So I'm opening 3-4 oldest indexes, and then starting to snapshot them. So when snapshot finishes, i'm deleting these indexes and then rebooting elastic and logstash. I also noticed if I\m not delete these indexes. services may not start. I mean they start, but messages not collecting. ☹ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:11am UTC](https://discuss.elastic.co/t/es-snapshots/1281/12 "2017-07-06T00:11:26Z")