# Elastic Search Query performance when source is disabled **URL:** \ **Category:** Elasticsearch\ **Created:** [October 24, 2022, 6:37pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348 "2022-10-24T18:37:42Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![prateek\_shekhar](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@prateek\_shekhar](https://discuss.elastic.co/u/prateek_shekhar)\ **Post date:** [October 24, 2022, 6:37pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/1 "2022-10-24T18:37:42Z")
Hi Everyone, we have an Elasticsearch index with 100 million documents (with replicas it's about 400 million). The index contains nested documents as well. We have a use case where we have to boost the score of the documents using some fields present in the document. For this boosting we are using function score query. Our response time when we disable the fetch operation is less than 30ms. We use this endpoint to disable the fetch ```auto https:///elastic_index_name/_search?_source=false ``` However when we enable the fetch the same response time becomes greater than 2 seconds. We tried to debug using the profile API, but based on the docs it doesn't look like the profile api returns the time spent during the fetch operation. Hence the output of the profile api shows time in milliseconds which is the same when we run the query with \_source disabled. We tried to use other forms of scoring like rankFeatures and script score query. But we haven't had any luck. Can someone please share if they have some insights into this issue? Please let me know if I any more details are needed from my end. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 26, 2022, 4:19am UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/2 "2022-10-26T04:19:09Z")
What sort of storage do your nodes use? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 26, 2022, 4:46am UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/3 "2022-10-26T04:46:02Z")
If you are using nested documents I assume your average document size might be reasonably large. Retrieving the source requires Elasticsearch to perform a lot of small reads across the file syste, and this will as Mark pointed out be impacted by how fast your storage is. ---
**Author:** ![prateek\_shekhar](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@prateek\_shekhar](https://discuss.elastic.co/u/prateek_shekhar)\ **Post date:** [October 26, 2022, 4:23pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/4 "2022-10-26T16:23:46Z")
Our storage is Amazon EBS. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 26, 2022, 4:31pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/5 "2022-10-26T16:31:12Z")
What type of EBS? How much storage do you have per node? How much data do you have per node? ---
**Author:** ![prateek\_shekhar](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@prateek\_shekhar](https://discuss.elastic.co/u/prateek_shekhar)\ **Post date:** [October 26, 2022, 4:42pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/6 "2022-10-26T16:42:54Z")
Our index size is about 17GB. Very few documents have very large number of nested docs in them. However for this use case we aren't searching or retrieving source information from the nested docs. Our query looks something like this: ```auto { "from": 0, "size": 10, "query": { "function_score": { "query": { "bool": { "must": [{ "match_all": { "boost": 1 } }], "filter": [ //Range and term filter queries ] } }, "functions": [ { // Functions.... } ] } } } ``` I checked the node\_cache stats as well for this query. Strangely the range and the term queries are not returned from the node cache here. We have another use case where the search is directed at some specific fields for some search keywords. Here the match\_all part in the bool must clause is replaced with multi\_match query on some fields. There we do some nested searches as well. However that is very fast (under 50ms). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 26, 2022, 5:28pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/7 "2022-10-26T17:28:59Z")
What type of EBS are you using? What is the size of the volume? ---
**Author:** ![prateek\_shekhar](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@prateek\_shekhar](https://discuss.elastic.co/u/prateek_shekhar)\ **Post date:** [October 26, 2022, 6:33pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/8 "2022-10-26T18:33:23Z")
We are using GP2 EBS volumes. Each node has about 100GB of storage space available and about 28GB of data. Our cluster size is 7 nodes (3 Master, 4 Data Nodes). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 26, 2022, 6:38pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/9 "2022-10-26T18:38:26Z")
Small gp2 EBS volumes fo not necessarily support a lot of IOPS. Check IO when you are querying using e.g. iostat to see if it is the storage that is slowing fown the response. ---
**Author:** ![prateek\_shekhar](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@prateek\_shekhar](https://discuss.elastic.co/u/prateek_shekhar)\ **Post date:** [October 26, 2022, 8:35pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/10 "2022-10-26T20:35:09Z")
I was able to grab some IO statistics using the \_nodes/stats API. I am not sure how to read these stats to find the bottleneck: Master Node: ```auto { "fs" : { "timestamp" : 1666816065822, "total" : { "total_in_bytes" : 7397171200, "free_in_bytes" : 4712013824, "available_in_bytes" : 4695236608 }, "data" : [ { "type" : "ext4", "total_in_bytes" : 7397171200, "free_in_bytes" : 4712013824, "available_in_bytes" : 4695236608 } ], "io_stats" : { "devices" : [ { "operations" : 105227556, "read_operations" : 26983, "write_operations" : 105200573, "read_kilobytes" : 480896, "write_kilobytes" : 1131588120 } ], "total" : { "operations" : 105227556, "read_operations" : 26983, "write_operations" : 105200573, "read_kilobytes" : 480896, "write_kilobytes" : 1131588120 } } } } ``` Data Node: ```auto { "fs" : { "timestamp" : 1666816065822, "total" : { "total_in_bytes" : 105554829312, "free_in_bytes" : 76648398848, "available_in_bytes" : 71262912512 }, "data" : [ { "type" : "ext4", "total_in_bytes" : 105554829312, "free_in_bytes" : 76648398848, "available_in_bytes" : 71262912512 } ], "io_stats" : { "devices" : [ { "operations" : 774302927, "read_operations" : 78, "write_operations" : 774302849, "read_kilobytes" : 312, "write_kilobytes" : 12612589788 } ], "total" : { "operations" : 774302927, "read_operations" : 78, "write_operations" : 774302849, "read_kilobytes" : 312, "write_kilobytes" : 12612589788 } } } } ``` Ours is read intensive cluster. I am not sure why the read\_operations here is more than the write\_operations. Is that the other way around, like: write\_operations refers to the fetch phase? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 27, 2022, 6:15am UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/11 "2022-10-27T06:15:02Z")
Run the command `iostat -x` from the command line on the host when queries are slow. This should show you if storage might be an issue. ---
**Author:** ![prateek\_shekhar](https://avatars.discourse-cdn.com/v4/letter/p/c89c15/32.png) [@prateek\_shekhar](https://discuss.elastic.co/u/prateek_shekhar)\ **Post date:** [October 28, 2022, 6:14pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/12 "2022-10-28T18:14:40Z")
I think disk IOPS was the bottleneck for us. We had some very heavy documents (about 35 - 40MB) in size. Their retrieval was really slow. What we ended up doing was to disable their storage (nested fields) from the \_source field. We didn't really need them to be returned in the response but needed them in the index for some scoring. Thank you for helping us with this issue. Much appreciated! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 25, 2022, 6:15pm UTC](https://discuss.elastic.co/t/elastic-search-query-performance-when-source-is-disabled/317348/13 "2022-11-25T18:15:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.