# High latency queries appear every 20 minutes

**URL:** <https://discuss.elastic.co/t/high-latency-queries-appear-every-20-minutes/249095>\
**Category:** Elasticsearch\
**Created:** [September 18, 2020, 12:28pm UTC](https://discuss.elastic.co/t/high-latency-queries-appear-every-20-minutes/249095 "2020-09-18T12:28:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\
**Post date:** [September 18, 2020, 12:28pm UTC](https://discuss.elastic.co/t/high-latency-queries-appear-every-20-minutes/249095/1 "2020-09-18T12:28:16Z")

</div>

[ES performance reliablity issue](https://discuss.elastic.co/t/es-performance-reliablity-issue/247974)  
After I fix the issue above, I find another strange issue.

The process of our es query is:  
[user application] -\> [nginx load balance] -\> [coordinating nodes] -\> [data nodes]

The search latency of most queries is less than 50ms. But there are still some queries which take more than 200ms in nginx logs.  
If I filter the log by upstream\_addr, I can find that each coordinating node has queries like this every 20 minutes. For example, the issue occurs on node A in the 13th、33rd、53rd minutes, and on node B in the 5th、25th、45th minutes.  
If I replace the 8C32G coordinating node with a 16C32G server, the number of high latency queries reduce to 1/2.  
I guess that the issue is related to search thread pool, because the thread pool size of the 16core ES node is larger, so the scope of one thread issue is smaller.But I can't prove it, I want to know what happened.

---

<div class="post-metadata">

**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\
**Post date:** [September 22, 2020, 2:13pm UTC](https://discuss.elastic.co/t/high-latency-queries-appear-every-20-minutes/249095/2 "2020-09-22T14:13:09Z")

</div>

I use the following APIs to list all default settings.  
\_cluster/settings?include\_defaults  
\_settings?include\_defaults

And find two items which have the default value 20m  
xpack.security.authc.token.timeout  
xpack.security.authz.store.roles.index.cache.ttl

---

<div class="post-metadata">

**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\
**Post date:** [September 24, 2020, 6:57am UTC](https://discuss.elastic.co/t/high-latency-queries-appear-every-20-minutes/249095/3 "2020-09-24T06:57:29Z")

</div>

I confirm that the native realm cache causes the issue.  
If I call this API \_xpack/security/realm/\*/\_clear\_cache ,the same issue orccurs immediatly.  
Now I do this setting(ES 5.6.3) to reduces the number of high latency queries:

xpack.security.authc.realms:  
realm1:  
type: native  
order: 0  
cache.ttl: 24h

But I don't know how realm cache causes the issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 22, 2020, 6:57am UTC](https://discuss.elastic.co/t/high-latency-queries-appear-every-20-minutes/249095/4 "2020-10-22T06:57:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.