# Elasticsearch queue issue after upgrading from 8.6.2 to 8.12.1/8.12.2 **URL:** \ **Category:** Elasticsearch\ **Created:** [March 8, 2024, 6:10pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052 "2024-03-08T18:10:08Z")\ **Posts on this page:** 17\ **Page:** 2
**Author:** ![Amos66](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@Amos66](https://discuss.elastic.co/u/Amos66)\ **Post date:** [March 10, 2024, 4:43pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/21 "2024-03-10T16:43:45Z")
the query with 0 fail. So no profiling. For me the problems is solved. I have a workaround. With other index I have similar problem. 6sec vs 44ms. I am sure it can be preproduce with any large index. > [@Ignacio\_Vera](#): > > I wonder if it would be possible to share the output of the nodes [hot threads](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) while running the query with `min_doc_count: 0`. It should tell us where we are expending those CPU cycles. How can I share here big file? ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 10, 2024, 5:07pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/22 "2024-03-10T17:07:28Z")
For big text files we normally recommend something like [github gist](https://docs.github.com/en/get-started/writing-on-github/editing-and-sharing-content-with-gists/creating-gists) ---
**Author:** ![elastic\_dude](https://avatars.discourse-cdn.com/v4/letter/e/f14d63/32.png) [@elastic\_dude](https://discuss.elastic.co/u/elastic_dude)\ **Post date:** [March 10, 2024, 6:13pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/23 "2024-03-10T18:13:18Z")
> That's true, but it's not obviously related - it's doing the same amount of work either way. Instead it sounds like there's been an unexpected regression in the performance of a certain kind of query. It'd be very useful if you could pin it down so we can open a bug report. I expect that enabling the [query slow log](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html) should help, but eliminating clients one-by-one is a reliable approach too. The tricky part here is we have not seen this happen in any of our dev or stage clusters and we are behind a GraphQL use case where we have hundreds of clients that each can send a few unique queries each (with changing parameters) and we can't just turn off the clients. We haven't been able to force the issue and stage and obviously don't want to put prod back to a scenario where we impact clients. We have slow query logging on the calling API side and in our non-comprehensive analysis (there are thousands of logged queries during these queue timings) we didn't notice anything abnormal or any spike in volume above normal. Another oddity is we only noticed it would start happening after 4-5 hours of running fine regardless of when we turned traffic back on. Back on 8.6.2 in the region/cluster we saw impact on, other clusters in that region are on 8.12.2 still and we are really hoping we can help find the root cause here to help get a fix out because restoring those clusters (with many many datasets that require intricate execution to keep fully matched with our other region/datastores upon restore) would be a huge undertaking. One other note, the impacted region's cluster only has one large (multi TB) index in it and the shards are above the normal \<50GB recommendation, they are at ~85 GB right now, just thought I would bring that up as a potential clue as I assume those shards will have more/larger segments than most of our indices in other clusters. ---
**Author:** ![Amos66](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@Amos66](https://discuss.elastic.co/u/Amos66)\ **Post date:** [March 11, 2024, 3:45am UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/24 "2024-03-11T03:45:37Z")
This is my hot threads result. When the query is active. My index has 10 shards, about 50 segments per shard and avg size 45G I cannot do too many tests on production cluster. [https://gist.github.com/AmosIturan/8d5fa73fdeb0f821616f5cc6a33e0a8a](https://gist.github.com/AmosIturan/8d5fa73fdeb0f821616f5cc6a33e0a8a) ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 11, 2024, 11:32am UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/25 "2024-03-11T11:32:38Z")
Thank you for the Hot threads output. I see you are running the query with the profiler set to true. I guess this is part of your investigation? The output confirms one of my suspicion as we see lots of them hitting this [method](https://github.com/elastic/elasticsearch/blob/8e6e0e59ea34be29a6495f0680944d6a7f559756/server/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/MapStringTermsAggregator.java#L374) ```auto app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313) app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389) app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247) app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111) app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75) ``` This method is collecting the zero doc buckets which iterates over the data in all segments. This means that if the query is being collected in parallel, we will be doing more work than before which might explain the behaviour you are describing. I have not being able to reproduce it locally because as you noted, volume is a factor but we believe this is not the right behaviour and we just merged a [PR](https://github.com/elastic/elasticsearch/pull/106156) that disables parallel collection when min\_doc\_count in the terms aggregation is 0, so we do the same work as before. ---
**Author:** ![Amos66](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@Amos66](https://discuss.elastic.co/u/Amos66)\ **Post date:** [March 11, 2024, 2:04pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/26 "2024-03-11T14:04:52Z")
> [@Ignacio\_Vera](#): > > I see you are running the query with the profiler set to true. I guess this is part of your investigation? Yes, It was running from Kibana devtools. And I left profile : true from previous test. > [@Ignacio\_Vera](#): > > I have not being able to reproduce it locally because as you noted, volume is a factor but we believe this is not the right behaviour and we just merged a [PR](https://github.com/elastic/elasticsearch/pull/106156) that disables parallel collection when min\_doc\_count in the terms aggregation is 0, so we do the same work as before. In my case because of the index size it actually made the service so slow that it was not really possible to work. and all nodes CPU raised to 100. It will probably affect others, and will be hard for them to know where this is coming from. ---
**Author:** ![elastic\_dude](https://avatars.discourse-cdn.com/v4/letter/e/f14d63/32.png) [@elastic\_dude](https://discuss.elastic.co/u/elastic_dude)\ **Post date:** [March 11, 2024, 5:22pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/27 "2024-03-11T17:22:55Z")
FWIW, I don't see any references of `min_doc_count` in our code base or logs so unfortunately I don't think that fix covers our regression. We don't do many aggregations in our use case. ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 11, 2024, 5:29pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/28 "2024-03-11T17:29:32Z")
Are you able to get hot threads and share it with us? I it should provide lots of information on your case. ---
**Author:** ![elastic\_dude](https://avatars.discourse-cdn.com/v4/letter/e/f14d63/32.png) [@elastic\_dude](https://discuss.elastic.co/u/elastic_dude)\ **Post date:** [March 11, 2024, 6:08pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/29 "2024-03-11T18:08:06Z")
At any time or are you only interested when the queues are there? ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 11, 2024, 6:29pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/30 "2024-03-11T18:29:13Z")
when the queues builds up so we can see where we are spending time, ---
**Author:** ![elastic\_dude](https://avatars.discourse-cdn.com/v4/letter/e/f14d63/32.png) [@elastic\_dude](https://discuss.elastic.co/u/elastic_dude)\ **Post date:** [March 11, 2024, 7:25pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/31 "2024-03-11T19:25:41Z")
That would require production impact to clients which is not something we certainly cannot due on purpose. We have also reverted the cluster we saw this issue to 8.6.2 at this time. ---
**Author:** ![Ignacio\_Vera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ignacio_vera/32/36674_2.png) [@Ignacio\_Vera](https://discuss.elastic.co/u/Ignacio_Vera)\ **Post date:** [March 12, 2024, 9:31am UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/32 "2024-03-12T09:31:48Z")
I understand. It is going to be difficult to understand your case, you might want to try to run hot threads even with the current system when it is busy. Maybe it shows some clues. ---
**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\ **Post date:** [March 28, 2024, 4:47pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/33 "2024-03-28T16:47:21Z")
Hey all, for future readers: we have merged a fix for this problem, see [Disable parallel collection for terms aggregation with min\_doc\_count equals to 0 by iverase · Pull Request #106156 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/106156) . ---
**Author:** ![elastic\_dude](https://avatars.discourse-cdn.com/v4/letter/e/f14d63/32.png) [@elastic\_dude](https://discuss.elastic.co/u/elastic_dude)\ **Post date:** [March 28, 2024, 5:09pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/34 "2024-03-28T17:09:55Z")
Just to clarify, that fix does not (AFAIK) address the issue I reported. It addresses the issue that Amos66 reported but does not cover the issue we have been facing, again AFAIK. ---
**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\ **Post date:** [March 28, 2024, 6:35pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/35 "2024-03-28T18:35:51Z")
Thanks for clarifying. I did not realize that the thread included two separate issues with similar symptoms. We'd love to get more details about the request that is triggering the high CPU usage, like we did for the min\_doc\_count scenario. ---
**Author:** ![elastic\_dude](https://avatars.discourse-cdn.com/v4/letter/e/f14d63/32.png) [@elastic\_dude](https://discuss.elastic.co/u/elastic_dude)\ **Post date:** [March 28, 2024, 6:51pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/36 "2024-03-28T18:51:03Z")
We would too, unfortunately we don't have a repeatable way to make that occur without impacting production traffic. We have several clusters that were updated to 8.12.x and we only see severe issues on one with medium sized infra/high traffic but see some smaller impact on large sized infra/high traffic so wondering if we are "brute forcing" our way out of impact there. Impacted cluster was rebuilt on 8.6.2, no issues since, we are trying to go to 8.9.2 on that cluster next week. Only lead I have right now is I noticed some queries that should be extremely performant/are super simple running quite slowly in updated cluster vs non-updated cluster that has the same exact data. Will dig into that more Thank you ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 25, 2024, 6:51pm UTC](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052/37 "2024-04-25T18:51:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/elasticsearch-queue-issue-after-upgrading-from-8-6-2-to-8-12-1-8-12-2/355052.md?page=1)