Xpack Unable to find cpuacct.usage

[sgarap]

Hi,

I tried installing X-Pack 5.2.0 to Elasticsearch 5.2.0 using docker and kubernetes. The following exception is being written to log files continuously. The file is actually has read permissions for every user. Is Elasticserach/xpack trying to modify this file for getting Node Statistics?

-rw-r--r--. 1 root root 0 Feb 15 10:00 cpuacct.usage

[DEBUG][o.e.m.o.OsProbe ] error reading control group stats
java.nio.file.NoSuchFileException: /sys/fs/cgroup/cpuacct/system.slice/docker-5cf7663bf515147fac2eed2d4eca5d4bbb314400c8d71d17dc5c9c7e386a434c.scope/cpuacct.usage
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_102]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_102]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_102]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_102]
at java.nio.file.Files.newBufferedReader(Files.java:2784) ~[?:1.8.0_102]
at java.nio.file.Files.readAllLines(Files.java:3202) ~[?:1.8.0_102]
at java.nio.file.Files.readAllLines(Files.java:3242) ~[?:1.8.0_102]
at org.elasticsearch.monitor.os.OsProbe.readSingleLine(OsProbe.java:185) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:274) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:261) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:419) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:466) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsService$OsStatsCache.refresh(OsService.java:64) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsService$OsStatsCache.refresh(OsService.java:57) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsService.stats(OsService.java:54) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:108) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:98) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:53) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:145) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:270) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:266) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:237) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:289) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$null$2(ServerTransportFilter.java:164) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:127) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:121) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:109) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:166) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:182) ~[x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$2(AuthenticationService.java:201) ~[x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:213) [x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:180) [x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:142) [x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:114) [x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:142) [x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:296) [x-pack-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1488) [elasticsearch-5.2.0.jar:5.2.0]

[spinscale (Alexander Reelsen)]

soooooo

after talking to a colleague, he mentioned that this is an issue in runc, that needs to be fixed in runc, then in docker, then all is good. See https://github.com/opencontainers/runc/pull/1184

first things first: This will just spill a non-fatal message, flooding the logs, but everything should still work.

However you could start the ES node with a system property -Des.cgroups.hierarchy.override=/ (note, system property, not ES parameter), to make this work again. Not that this is an undocumented parameter, that could be removed or changed over time.

Hope this helps.

--Alex

[sgarap]

Thanks for the help Alex. Adding the mentioned System property stopped the log messages