# Identify indexing performance bottleneck
**URL:** https://discuss.elastic.co/t/identify-indexing-performance-bottleneck/252456
**Category:** Kibana
**Created:** [October 18, 2020, 4:47pm UTC](https://discuss.elastic.co/t/identify-indexing-performance-bottleneck/252456 "2020-10-18T16:47:58Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@larryzhu](https://discuss.elastic.co/u/larryzhu)
#### Post date: [October 18, 2020, 4:47pm UTC](https://discuss.elastic.co/t/identify-indexing-performance-bottleneck/252456/1 "2020-10-18T16:47:58Z")
we have elastic data nodes that run on 52 physical processors/104 logic processors, the machines are using Intel Xeon Platinum 8167M. Base frequency 2.0 GHz, max turbo frequency 2.4 GHz.
however we are consistently seeing the machine running over 90% CPU with load above 100. this is odd as we expect it is bounded by disk I/O speed while the disk load is consistently low suggesting we cannot push data to disk fast enough and the CPU is the bottleneck.
Question: how do we profile ES data nodes and see what are the hotspots and why the CPUs are busy?
we are running elastic search version 7.8.1
---
### Author:  [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [October 18, 2020, 5:31pm UTC](https://discuss.elastic.co/t/identify-indexing-performance-bottleneck/252456/2 "2020-10-18T17:31:10Z")
Use your favorite "top" to make sure that the CPU is being used by elastic processes, then use the hot threads api [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html)
---
### Author:  [@larryzhu](https://discuss.elastic.co/u/larryzhu)
#### Post date: [October 18, 2020, 8:34pm UTC](https://discuss.elastic.co/t/identify-indexing-performance-bottleneck/252456/3 "2020-10-18T20:34:22Z")
I got the results back below. Can you please shed lights we can turn off or minimizing the RBACEngine costs as shown here?
Top command:
![]()
`GET _nodes/scs-c-prod-iad-adp-elk-hot-1/hot_threads`
```auto
::: {scs-c-prod-iad-adp-elk-hot-1}{ADIB-a1GQRK4Y9EMbfypqw}{MA1pStFqRFeLxjaCF-qCjg}{10.128.1.54}{10.128.1.54:9300}{dir}{data=hot, xpack.installed=true, transform.node=false}
Hot threads at 2020-10-18T19:46:02.829Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
111.4% (556.9ms out of 500ms) cpu usage by thread 'elasticsearch[scs-c-prod-iad-adp-elk-hot-1][transport_worker][T#50]'
2/10 snapshots sharing following 268 elements
[java.base@14.0.1/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.hasNext](mailto:java.base@14.0.1/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.hasNext)(Collections.java:1676)
org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:507)
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:345)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:255)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4807/0x00007f18d1fca108.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:663)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:258)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4808/0x00007f18d1fc9840.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:663)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:306)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4811/0x00007f18d1fc4c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:330)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:303)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:269)
org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:234)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:199)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4588/0x00007f1c822c7960.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:121)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4590/0x00007f1c822c5908.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:250)
org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:127)
org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:115)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:201)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4803/0x00007f18d1fcd160.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4511/0x00007f1c822014b0.accept(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:385)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4512/0x00007f1c822708b0.run(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:396)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156)
org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108)
app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)
app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.handleOperationRequest(TransportReplicationAction.java:251)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$$Lambda$3453/0x00007f1ca86674b0.messageReceived(Unknown Source)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$4507/0x00007f1c82204508.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:352)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4824/0x00007f18d1ae94b0.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4489/0x00007f1c8220f4b0.accept(Unknown Source)
[java.base@14.0.1/java.util.ArrayList.forEach](mailto:java.base@14.0.1/java.util.ArrayList.forEach)(ArrayList.java:1510)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:63)
org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:347)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4822/0x00007f18d1ae7d68.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4489/0x00007f1c8220f4b0.accept(Unknown Source)
[java.base@14.0.1/java.util.ArrayList.forEach](mailto:java.base@14.0.1/java.util.ArrayList.forEach)(ArrayList.java:1510)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
```
---
### Author:  [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [October 20, 2020, 1:05am UTC](https://discuss.elastic.co/t/identify-indexing-performance-bottleneck/252456/4 "2020-10-20T01:05:48Z")
Related post maybe [CPU at 100% with XPACK security Enabled - ES 7.3](https://discuss.elastic.co/t/cpu-at-100-with-xpack-security-enabled-es-7-3/214544)
How may indices, shards, replicas? Are you using enhanced access, field or document level checks?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 17, 2020, 1:05am UTC](https://discuss.elastic.co/t/identify-indexing-performance-bottleneck/252456/5 "2020-11-17T01:05:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.