# High Request Overhead Not Captured by Took **URL:** \ **Category:** Elasticsearch\ **Created:** [October 18, 2019, 8:09pm UTC](https://discuss.elastic.co/t/high-request-overhead-not-captured-by-took/204275 "2019-10-18T20:09:02Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Zach\_Diemer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_diemer/32/56182_2.png) [@Zach\_Diemer](https://discuss.elastic.co/u/Zach_Diemer)\ **Post date:** [October 18, 2019, 8:09pm UTC](https://discuss.elastic.co/t/high-request-overhead-not-captured-by-took/204275/1 "2019-10-18T20:09:02Z")
Hey all! I've been trying to nail down where I can improve performance of my Elasticsearch cluster, and I've been specifically attempting to find out more about what is causing some high overhead in my end-to-end request. I'm using Elasticsearch 6.4.2 and my client is [Elasticsearch.Net](http://Elasticsearch.Net) 6.3.1 (NEST in .NET). I have my cluster deployed as a GKE deployment with six nodes. I have 11 indices, each with a single shard and 5 replicas. None of my indices are larger than about 4GB. I'm capturing end-to-end request latency as well as the took parameter that Elastic reports back. Took is regularly only 50% of the end-to-end request latency. End-to-end is roughly 600ms on average. Even when I query using size: 0, I'm still seeing that overhead. I've done a pod-to-pod request within my Kubernetes cluster and found that the network latency is roughly 15-40ms, so I've ruled out network latency. My NEST queries are quite long due to the fact that I'm doing a complex faceted multi-match query across my 11 indices. This requires a bunch of aggregations, filters, and post-filters. Additionally, I'm including a highlighter and score functions. An average NEST query turns out to about 800 lines of JSON. Is this overhead really only due to serialization? Is there any way I can pare this down? ---
**Author:** ![Zach\_Diemer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zach_diemer/32/56182_2.png) [@Zach\_Diemer](https://discuss.elastic.co/u/Zach_Diemer)\ **Post date:** [October 21, 2019, 6:10pm UTC](https://discuss.elastic.co/t/high-request-overhead-not-captured-by-took/204275/2 "2019-10-21T18:10:18Z")
Hey all, just wanted to include this chart that shows a breakdown in our end-to-end request latency so y'all can see what I'm describing. `search-took` is the `took` value that Elasticsearch reports back, whereas `search-query-ms-excluding-took` is purely the `await ElasticClient.SearchAsync` method minus the `took` value. (`search-handler-ms-excluding-query` and `search-latency-web-full` are additional overhead where we're pulling data from our database, etc.) ![29%20AM](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fabf1e09e155fde960601b30ca418f87eaaab072.png) You can see how the `search-query-ms-excluding-took` regularly _exceeds_ `took`. What's going on? ---
**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\ **Post date:** [October 21, 2019, 7:17pm UTC](https://discuss.elastic.co/t/high-request-overhead-not-captured-by-took/204275/3 "2019-10-21T19:17:35Z")
Hi @Zach_Diemer, > [@Zach\_Diemer](#): > > I've done a pod-to-pod request within my Kubernetes cluster and found that the network latency is roughly 15-40ms, so I've ruled out network latency. > > My NEST queries are quite long due to the fact that I'm doing a complex faceted multi-match query across my 11 indices. This requires a bunch of aggregations, filters, and post-filters. Additionally, I'm including a highlighter and score functions. An average NEST query turns out to about 800 lines of JSON. I think the problem here is the combination of relatively high latency in your cluster and complicated queries. Assuming you have an average latency of `30ms` and you're seeing `300ms`, then 60ms of that is probably already explained by network latency (since there's quite some time between the server receiving the request and responding I'm estimating the latency as two separate network request approximately). Sending 800 lines of JSON and deserialising def will take non-trivial time on the server so that likely is another chunk of your wall-time. Also, you client will take non-trivial time to serialise those queries as well. The same then goes for the query response (it has to be serialised on ES end and deserialised by the client). One way of gaining some insight into the latency serialisation introduces into the end-to-end latency may be to record a few queries as JSON strings/files and running them via `curl` (or some other REST client) to isolate the time cost of serialising the query on the client? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 18, 2019, 7:17pm UTC](https://discuss.elastic.co/t/high-request-overhead-not-captured-by-took/204275/4 "2019-11-18T19:17:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.