I see a bunch of posts about high CPU utilization but after reading a few i don't
see a remedy.
I installed filebeat on 5 OpenStack servers and put ELK Stack (5.2.2) on a 6th machine which also
serves as a login server running XRDP so that i can use the OpenStack thru RDP from
windows. Elasticsearch after instaling ELK is hard to connect with RDP (works about 1 time in
10). I assume this is because of elastic high CPU utilization (always over 100%).
The login server (with ELK stack) is dual Xeon with 12 physical cores (no hyper threading).
I attached capture of 'top' showing CPU utilization .
I added some of the elasticsearch hot_threads output below. editor would let me add all
hot_threads output.
Is it bad practice to install elastic search on a machine that servers RDP? Is there a way to
reduce elastic search CPU utilization? What is the latest thinking on managing elastic
CPU% ?
thx,
Rod
here is elasticsearch output
{
"name" : "NZK5rQN",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "pQmLGu2oSdqPz6T7JgUFeA",
"version" : {
"number" : "5.2.2",
"build_hash" : "f9d9b74",
"build_date" : "2017-02-24T17:26:45.835Z",
"build_snapshot" : false,
"lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}
-------------------------------- hot_ threads output ---------------------------------------------------------
::: {NZK5rQN}{NZK5rQNsSDi1pklwhr9kXw}{LrRHfoieRCaYd04Zw_Zp7g}{localhost}{[::1]:9300}
Hot threads at 2017-03-31T14:39:11.761Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
84.0% (419.8ms out of 500ms) cpu usage by thread 'elasticsearch[NZK5rQN][management][T#5]'
3/10 snapshots sharing following 56 elements
java.security.AccessController.getStackAccessControlContext(Native Method)
java.security.AccessController.checkPermission(AccessController.java:858)
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1373)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1362)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1349)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:301)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:703)
org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:177)
org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:104)
org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:53)
org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:145)
org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:270)
org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:266)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:237)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:289)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$1538/1903947112.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$null$2(ServerTransportFilter.java:164)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile$$Lambda$1544/278447279.accept(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:127)
org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:121)
org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:109)