# Query performance drop while doing a snapshot **URL:** \ **Category:** Elasticsearch\ **Created:** [January 25, 2018, 1:42pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052 "2018-01-25T13:42:32Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Alexander\_Iliev](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Alexander\_Iliev](https://discuss.elastic.co/u/Alexander_Iliev)\ **Post date:** [January 25, 2018, 1:42pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/1 "2018-01-25T13:42:32Z")
We are running a 6-node Elasticsearch cluster with a single (not really single, but the only one that is actively used) index which is 3.5TB in size (51 shards, 25 primary). We have a NFS share mounted on all nodes and registered as a snapshot repository. We triggered a snapshot (the first one), but the query performance degraded extremely during the snapshot so we canceled the snapshot. My guess is that this is expected, but what can we do to avoid or at least reduce this search performance drop? Thank you! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 25, 2018, 2:18pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/2 "2018-01-25T14:18:48Z")
That's not supposed to be happening. Did you change any default settings ? ---
**Author:** ![Alexander\_Iliev](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Alexander\_Iliev](https://discuss.elastic.co/u/Alexander_Iliev)\ **Post date:** [January 25, 2018, 2:26pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/3 "2018-01-25T14:26:52Z")
Do you have any particular settings in mind? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 25, 2018, 2:39pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/4 "2018-01-25T14:39:33Z")
Yes. But share what you have. Node settings and index settings please. ---
**Author:** ![Alexander\_Iliev](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Alexander\_Iliev](https://discuss.elastic.co/u/Alexander_Iliev)\ **Post date:** [January 25, 2018, 3:21pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/5 "2018-01-25T15:21:11Z")
Here's the [output](https://pastebin.com/LKX8Q4cm) from the `GET /_nodes/stats` and `GET /index_name/_settings` API calls. If those are not the settings you had in mind please tell me the exact calls you want to see the output from. Thanks! (Note that the node, cluster and index names have been sanitized for privacy purposes.) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 25, 2018, 3:42pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/6 "2018-01-25T15:42:29Z")
Can you share your `elasticsearch.yml` file? ---
**Author:** ![Alexander\_Iliev](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Alexander\_Iliev](https://discuss.elastic.co/u/Alexander_Iliev)\ **Post date:** [January 25, 2018, 3:45pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/7 "2018-01-25T15:45:41Z")
Here you go: ```bash # cat /etc/elasticsearch/elasticsearch.yml | grep -vE '^#' cluster.name: elastic node.name: node-3 path.data: /data/elastic,/elastic path.logs: /var/log/elasticsearch path.repo: ["/mnt/elasticsearch/snapshot/e1/es-v5"] network.host: _eth1_,_local_ discovery.zen.ping.unicast.hosts: ["172.31.31.10","172.31.31.21","172.31.31.23","172.31.31.24","172.31.31.25"] discovery.zen.minimum_master_nodes: 4 ``` The rest of the nodes are configured similarly (the node name obviously differs for example). ---
**Author:** ![Alexander\_Iliev](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Alexander\_Iliev](https://discuss.elastic.co/u/Alexander_Iliev)\ **Post date:** [January 25, 2018, 3:46pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/8 "2018-01-25T15:46:54Z")
Just to clarify - all nodes are at version 5.6.1. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 25, 2018, 5:27pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/9 "2018-01-25T17:27:29Z")
I don't see anything obvious. Could you run hot\_threads API while the snapshot is running ? May be we can find something. Cc @Igor_Motov ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 25, 2018, 5:34pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/10 "2018-01-25T17:34:20Z")
What type of storage do you have? What does disk I/O and iowait look like when you are snapshotting compared to when you are not? ---
**Author:** ![Alexander\_Iliev](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Alexander\_Iliev](https://discuss.elastic.co/u/Alexander_Iliev)\ **Post date:** [January 25, 2018, 7:41pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/11 "2018-01-25T19:41:04Z")
I can't start a snapshot right now. I will post an update when we are able to do a snapshot (tomorrow most probably). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [January 26, 2018, 2:53pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/12 "2018-01-26T14:53:09Z")
The snapshot process should be throttled enough in order to not affect querying that much. Unless the throttling settings were change or the number of snapshot threads was changed in the thread pool settings, there shouldn't be a visible impact. We also need to consider, that even with throttling snapshot definitely adds some disk I/O, network and a little bit of memory load so if any nodes in the cluster were already very high in resource utilization, it is possible for a snapshot to be the last drop that triggers overload. I saw it a few times with S3 repo, that used to use memory for buffering, but it's not common for shared file system. So, I am a bit puzzled here. @tlrx any other thoughts? ---
**Author:** ![Alexander\_Iliev](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Alexander\_Iliev](https://discuss.elastic.co/u/Alexander_Iliev)\ **Post date:** [January 26, 2018, 5:48pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/13 "2018-01-26T17:48:17Z")
@dadoonet [Here](https://pastebin.com/ZqThEjy0)'s the hot\_threads API response. @Christian_Dahlqvist, the IO wait on the nodes look reasonable during the snapshot - most of the time below 20% (with some peaks, but below 70%). @Igor_Motov, that's what I expected, thanks for the clarification. The cluster was far from its limits at the time of the snapshot, so not sure why this happened. Anyway, a snapshot is in progress right now and the performance looks normal. I'll keep monitoring the service. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 23, 2018, 5:48pm UTC](https://discuss.elastic.co/t/query-performance-drop-while-doing-a-snapshot/117052/14 "2018-02-23T17:48:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.