# \[7.14.1\] Linearly increasing latency in task manager index searches **URL:** \ **Category:** Kibana\ **Created:** [October 6, 2021, 10:33am UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010 "2021-10-06T10:33:20Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![spiqueras](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spiqueras/32/95498_2.png) [@spiqueras](https://discuss.elastic.co/u/spiqueras)\ **Post date:** [October 6, 2021, 10:33am UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010/1 "2021-10-06T10:33:20Z")
We're migrating from a 6.8 cluster to a 7.14 cluster and we're seeing some abnormal behavior with Kibana's task manager index. Sorry if this is the wrong forum. For context, this is a 3-node ES v7.14.1 cluster that as of now we're only writing data to. We noticed that the Search Latency of some nodes was increasing linearly, despite having no clients doing any searches. After checking the indices, we pinned it down to Kibana's Task Manager index. These are the index metrics for the last week: ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/6/e604690e9bf6ba5378388ccb4af5620c3c37041b.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e937afb335fd6a698aeec860154843db0b113d8.png) We also see that the index size right now takes over 200MB, and it keeps growing: ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/d/7d883928958260b1e85208827784b878f150d412.png) Using cat api, we see that the index has 15 docs and more than 600k pending deletes. I guess those deletes are the reason why the searches take progressively longer? ```auto green open .kibana_task_manager_7.14.1_001 UUID 1 1 15 628919 270.3mb 135.1mb ``` We could force merge to remove the deleted docs, but it seems a bit silly that an index with 15 docs is taking up over 270 MB of disk, with search requests taking over 100ms. We'd like to know whether a) this is expected behavior and b) whether there's any setting that can be adjusted to improve the performance of this index, if needed. It might not be an issue but we haven't seen anything similar in our current ES 6.8 cluster and we want to make sure we're gtg before migrating the search clients. ---
**Author:** ![ying.mao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ying.mao/32/88151_2.png) [@ying.mao](https://discuss.elastic.co/u/ying.mao)\ **Post date:** [October 7, 2021, 4:55pm UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010/2 "2021-10-07T16:55:21Z")
HI @spiqueras, This does not sound like expected behavior. Can you provide the output of the [Task manager health API](https://www.elastic.co/guide/en/kibana/7.14/task-manager-api-health.html)? That will provide us with an idea of the task types that are running. We did add a cleanup in 7.14 that deletes up task documents from failed actions (related to rules and connectors, sometimes these failed actions could build up prior to 7.14) on upgrade, but since you are upgrading from 6.8, it doesn't seem like you should have any rules running. Can you confirm whether or not you are using rules and connectors? Thanks, ---
**Author:** ![spiqueras](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spiqueras/32/95498_2.png) [@spiqueras](https://discuss.elastic.co/u/spiqueras)\ **Post date:** [October 8, 2021, 9:05am UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010/3 "2021-10-08T09:05:42Z")
Hi @ying.mao , thanks for the quick response. You'll find the response of the Task Manager Health API in the [following gist](https://gist.github.com/spiqueras/88d393219761d22a00c3efc776bdb542). Regarding your comment, sorry if I wasn't clear. We're not upgrading the 6.8 cluster directly, we're running both clusters in parallel in different machines so we can test for regressions. Nevertheless, I can confirm we haven't setup any rules or connectors in the new cluster. In the meantime, we have added two more nodes to the cluster. This is how the Task Manager Index metrics look right now: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/8/38ec568143e5d315ae111397f15c3b760243e5c5.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/2/2289822378c72f064f126fd04dead9da8b518070.png) The discontinuities you see in the first chart roughly correspond to the moment where we added each of the nodes. Request time and latency go down, but later exhibit a similar increasing pattern. The index size in disk is also increasing. Other metrics of the same index show erratic behavior: ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/f/dfbd5f5b2477f353862f542dba2533df5724e96a.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/70d26a5cd8090fcc9865876567788ab5a940ffe0.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/e/be7a8bdae514ff215391db639599d1fad2a950fa.png) Now that I think of it, **we run a Kibana instance for each Elasticsearch node** (right now it's 5 ES nodes and 5 Kibana instances). This setup works well in our 6.8 cluster, but I wonder if the recent changes to Task Manager are somehow interfering with our setup. It's also a bit weird that the Task Manager Health API reports that the number of `observed_kibana_instances` is 1, when the interface reports 5 instances as expected, so there might be something misconfigured there. Thanks again for your time ---
**Author:** ![ying.mao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ying.mao/32/88151_2.png) [@ying.mao](https://discuss.elastic.co/u/ying.mao)\ **Post date:** [October 8, 2021, 12:46pm UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010/4 "2021-10-08T12:46:28Z")
Hi @spiqueras, Thanks for providing the task manager health output. Everything looks fairly normal to me. Is the task manager search latency affecting the rest of your cluster? Or is your concern driven primarily by the charts? The way task manager works is by claiming and running tasks at a specified poll interval (default 3 seconds). Every time it makes a claim or updates a task status (2-3 times per task per run), it updates the underlying document in the task manager index. Since updates in Elasticsearch are actually creating a new document and deleting the old, this accounts for the number of tomb-stoned documents you are seeing in the index. Task manager is the only feature in Kibana that uses the task manager index and the search latency, while increasing is still in the order of milliseconds, so I think as long as you are not seeing performance issues wrt the rest of your cluster, this is not something to be concerned about. Re: the # observed Kibana instances. This is an experimental capacity estimation we are trying out and it uses the server UUIDs that we see in the task manager index. Since you have very few tasks running, it seems that a single Kibana instance is picking up all the tasks so only 1 UUID is seen in the index, leading to the observed value of 1. Hope that helps! ---
**Author:** ![spiqueras](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spiqueras/32/95498_2.png) [@spiqueras](https://discuss.elastic.co/u/spiqueras)\ **Post date:** [October 11, 2021, 11:09am UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010/5 "2021-10-11T11:09:58Z")
Hi @ying.mao , Thanks for the explanation. It makes sense that the number of deleted documents is that high. It's a bit weird that ES/Lucene isn't claiming that space with more frequent merges, but we haven't tested whether this has a noticeable impact on the overall cluster performance (and we don't really have a baseline to compare against other than the old cluster, which is running on different hardware). We'll keep monitoring the performance as we move more clients to the new cluster and let you know if there's anything off. ---
**Author:** ![spiqueras](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spiqueras/32/95498_2.png) [@spiqueras](https://discuss.elastic.co/u/spiqueras)\ **Post date:** [October 20, 2021, 7:27am UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010/6 "2021-10-20T07:27:57Z")
As a follow-up, the problem has solved itself unexpectedly. We've started to take periodic snapshots with `include_global_state` set to `true`, and that has somehow caused ES to start reclaiming deleted documents more aggressively. Let's see it in data. All graphs correspond to the task manager index: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/8/a862f6c40ccb65fbfeec63d33463bfc7046cbade.png) The first snapshot was taken manually on Oct 18th 10am, and then in the afternoon we launched the periodic snapshotting process. You can see pinpoint those moments in the above chart by looking at the Doc Values line. Disk and Latency: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/c/ccfc44b84b2492efb1c325bef5aae1518b26a2d1.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/7/370fa971e8369ef0d4c1c1e7383d99c7d0decb9e.png) Primaries now take \<1MB in disk, whereas Search Latency is under 2ms, as expected from an index with 15 documents. To round it up, `/_cat/indices` output now shows a healthy number of deleted docs: `green open .kibana_task_manager_7.14.1_001 UUID 1 1 15 421 55.1mb 445.8kb` Thank you again for your time 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 17, 2021, 7:28am UTC](https://discuss.elastic.co/t/7-14-1-linearly-increasing-latency-in-task-manager-index-searches/286010/7 "2021-11-17T07:28:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.