Identify indexing performance bottleneck

larryzhu · October 18, 2020, 4:47pm

we have elastic data nodes that run on 52 physical processors/104 logic processors, the machines are using Intel Xeon Platinum 8167M. Base frequency 2.0 GHz, max turbo frequency 2.4 GHz.
however we are consistently seeing the machine running over 90% CPU with load above 100. this is odd as we expect it is bounded by disk I/O speed while the disk load is consistently low suggesting we cannot push data to disk fast enough and the CPU is the bottleneck.

Question: how do we profile ES data nodes and see what are the hotspots and why the CPUs are busy?

we are running elastic search version 7.8.1

rugenl · October 18, 2020, 5:31pm

Use your favorite "top" to make sure that the CPU is being used by elastic processes, then use the hot threads api https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

larryzhu · October 18, 2020, 8:34pm

I got the results back below. Can you please shed lights we can turn off or minimizing the RBACEngine costs as shown here?

Top command:

GET _nodes/scs-c-prod-iad-adp-elk-hot-1/hot_threads


::: {scs-c-prod-iad-adp-elk-hot-1}{ADIB-a1GQRK4Y9EMbfypqw}{MA1pStFqRFeLxjaCF-qCjg}{10.128.1.54}{10.128.1.54:9300}{dir}{data=hot, xpack.installed=true, transform.node=false}

 Hot threads at 2020-10-18T19:46:02.829Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:



 111.4% (556.9ms out of 500ms) cpu usage by thread 'elasticsearch[scs-c-prod-iad-adp-elk-hot-1][transport_worker][T#50]'

   2/10 snapshots sharing following 268 elements

[java.base@14.0.1/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.hasNext](mailto:java.base@14.0.1/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.hasNext)(Collections.java:1676)

     org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:507)

     org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:345)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:255)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4807/0x00007f18d1fca108.getAsync(Unknown Source)

     org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:663)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:258)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4808/0x00007f18d1fc9840.getAsync(Unknown Source)

     org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:663)

     org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:306)

     org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4811/0x00007f18d1fc4c40.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:330)

     org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:303)

     org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:269)

     org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:234)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:199)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4588/0x00007f1c822c7960.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)

     org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:121)

     org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4590/0x00007f1c822c5908.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:250)

     org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:127)

     org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:115)

     org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:201)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4803/0x00007f18d1fcd160.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4511/0x00007f1c822014b0.accept(Unknown Source)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:385)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4512/0x00007f1c822708b0.run(Unknown Source)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:396)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320)

     org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261)

     org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156)

     org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108)

     app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)

     app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155)

     app//org.elasticsearch.action.support.replication.TransportReplicationAction.handleOperationRequest(TransportReplicationAction.java:251)

     app//org.elasticsearch.action.support.replication.TransportReplicationAction$$Lambda$3453/0x00007f1ca86674b0.messageReceived(Unknown Source)

     org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257)

     app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

     app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)

     org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306)

     org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$4507/0x00007f1c82204508.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:352)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4824/0x00007f18d1ae94b0.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)

     app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

     app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4489/0x00007f1c8220f4b0.accept(Unknown Source)

[java.base@14.0.1/java.util.ArrayList.forEach](mailto:java.base@14.0.1/java.util.ArrayList.forEach)(ArrayList.java:1510)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)

     app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)

     app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)

     org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:63)

     org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)

     org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:347)

     org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4822/0x00007f18d1ae7d68.accept(Unknown Source)

     app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)

     app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

     app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4489/0x00007f1c8220f4b0.accept(Unknown Source)

[java.base@14.0.1/java.util.ArrayList.forEach](mailto:java.base@14.0.1/java.util.ArrayList.forEach)(ArrayList.java:1510)

     app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)

     app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)

rugenl · October 20, 2020, 1:05am

Related post maybe CPU at 100% with XPACK security Enabled - ES 7.3

How may indices, shards, replicas? Are you using enhanced access, field or document level checks?

system · November 17, 2020, 1:05am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
High CPU Utilisation in 8.11.4 Elasticsearch	7	134	April 9, 2024
How to inprove the cpu usage? Elasticsearch	6	383	June 16, 2020
How to fully utilize the resources of Elasticsearch node Elasticsearch	4	446	October 5, 2020
High CPU Usage on Data Node Elasticsearch	4	3115	July 5, 2017
Mismatched CPU usages on data nodes Elasticsearch	6	540	January 28, 2019

Identify indexing performance bottleneck

Related Topics