Identify indexing performance bottleneck

we have elastic data nodes that run on 52 physical processors/104 logic processors, the machines are using Intel Xeon Platinum 8167M. Base frequency 2.0 GHz, max turbo frequency 2.4 GHz.
however we are consistently seeing the machine running over 90% CPU with load above 100. this is odd as we expect it is bounded by disk I/O speed while the disk load is consistently low suggesting we cannot push data to disk fast enough and the CPU is the bottleneck.

Question: how do we profile ES data nodes and see what are the hotspots and why the CPUs are busy?

we are running elastic search version 7.8.1

Use your favorite "top" to make sure that the CPU is being used by elastic processes, then use the hot threads api https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

I got the results back below. Can you please shed lights we can turn off or minimizing the RBACEngine costs as shown here?

Top command:

GET _nodes/scs-c-prod-iad-adp-elk-hot-1/hot_threads


::: {scs-c-prod-iad-adp-elk-hot-1}{ADIB-a1GQRK4Y9EMbfypqw}{MA1pStFqRFeLxjaCF-qCjg}{10.128.1.54}{10.128.1.54:9300}{dir}{data=hot, xpack.installed=true, transform.node=false}

 Hot threads at 2020-10-18T19:46:02.829Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:



 111.4% (556.9ms out of 500ms) cpu usage by thread 'elasticsearch[scs-c-prod-iad-adp-elk-hot-1][transport_worker][T#50]'

   2/10 snapshots sharing following 268 elements

[java.base@14.0.1/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.hasNext](mailto:java.base@14.0.1/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.hasNext)(Collections.java:1676)

     org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:507)

     org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:345)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:255)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4807/0x00007f18d1fca108.getAsync(Unknown Source)

     org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:663)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:258)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4808/0x00007f18d1fc9840.getAsync(Unknown Source)

     org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:663)

     org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:306)

     org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4811/0x00007f18d1fc4c40.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:330)

     org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:303)

     org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:269)

     org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:234)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:199)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4588/0x00007f1c822c7960.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)

     org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:121)

     org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4590/0x00007f1c822c5908.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:250)

     org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:127)

     org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:115)

     org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:201)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4803/0x00007f18d1fcd160.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4511/0x00007f1c822014b0.accept(Unknown Source)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:385)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4512/0x00007f1c822708b0.run(Unknown Source)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:396)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261)

     org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108)

     app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)

     app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155)

     app//org.elasticsearch.action.support.replication.TransportReplicationAction.handleOperationRequest(TransportReplicationAction.java:251)

     app//org.elasticsearch.action.support.replication.TransportReplicationAction$$Lambda$3453/0x00007f1ca86674b0.messageReceived(Unknown Source)

     org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257)

     app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

     app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)

     org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306)

     org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$4507/0x00007f1c82204508.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:352)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4824/0x00007f18d1ae94b0.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)

     app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

     app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4489/0x00007f1c8220f4b0.accept(Unknown Source)

[java.base@14.0.1/java.util.ArrayList.forEach](mailto:java.base@14.0.1/java.util.ArrayList.forEach)(ArrayList.java:1510)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)

     app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)

     app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)

     org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:63)

     org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:347)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4822/0x00007f18d1ae7d68.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)

     app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

     app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4489/0x00007f1c8220f4b0.accept(Unknown Source)

[java.base@14.0.1/java.util.ArrayList.forEach](mailto:java.base@14.0.1/java.util.ArrayList.forEach)(ArrayList.java:1510)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)

     

Related post maybe CPU at 100% with XPACK security Enabled - ES 7.3

How may indices, shards, replicas? Are you using enhanced access, field or document level checks?