# ECK deployment fails on container runtime containerD **URL:** \ **Category:** Kibana\ **Tags:** docker\ **Created:** [September 29, 2021, 2:33pm UTC](https://discuss.elastic.co/t/eck-deployment-fails-on-container-runtime-containerd/285488 "2021-09-29T14:33:14Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![keshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keshara/32/63979_2.png) [@keshara](https://discuss.elastic.co/u/keshara)\ **Post date:** [September 29, 2021, 2:33pm UTC](https://discuss.elastic.co/t/eck-deployment-fails-on-container-runtime-containerd/285488/1 "2021-09-29T14:33:14Z")
Hi, I setup a Kubernetes cluster on vmware bare-metal environment. This Kubernetes cluster backed by container run-time =\> **containerD** (not Docker). ```auto # kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME eck-master Ready control-plane,master 46m v1.21.4 172.28.141.59 172.28.141.59 CentOS Linux 8 4.18.0-240.el8.x86_64 containerd://1.4.9 eck-worker1 Ready 44m v1.21.4 172.28.141.60 172.28.141.60 CentOS Linux 8 4.18.0-240.el8.x86_64 containerd://1.4.9 eck-worker2 Ready 44m v1.21.4 172.28.141.61 172.28.141.61 CentOS Linux 8 4.18.0-240.el8.x86_64 containerd://1.4.9 ``` Notice - The Container-Runtime =\> **containerd://1.4.9** - Also the Kubernetes cluster version =\> **v1.21.4** Now When I deployed the basic Elastic Cluster, I get to see Master pod throwing following Errors. ```auto Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:65) at java.base/jdk.internal.platform.Container.metrics(Container.java:43) at jdk.management/com.sun.management.internal.OperatingSystemImpl.(OperatingSystemImpl.java:48) at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(PlatformMBeanProviderImpl.java:279) at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(PlatformMBeanProviderImpl.java:198) at java.management/java.lang.management.ManagementFactory.lambda$getPlatformMBeanServer$0(ManagementFactory.java:487) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1766) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at java.management/java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:488) at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:140) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:558) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:207) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:220) at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:197) at org.elasticsearch.common.logging.LogConfigurator.configureStatusLogger(LogConfigurator.java:248) at org.elasticsearch.common.logging.LogConfigurator.configureWithoutConfig(LogConfigurator.java:95) at org.elasticsearch.cli.CommandLoggingConfigurator.configureLoggingWithoutConfig(CommandLoggingConfigurator.java:29) at org.elasticsearch.cli.Command.main(Command.java:76) at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:32) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:61) ... 26 more Caused by: java.lang.ExceptionInInitializerError at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:107) at java.base/jdk.internal.platform.CgroupMetrics.getInstance(CgroupMetrics.java:167) ... 31 more Caused by: java.lang.NullPointerException at java.base/java.util.Objects.requireNonNull(Objects.java:208) at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:260) at java.base/java.nio.file.Path.of(Path.java:147) at java.base/java.nio.file.Paths.get(Paths.java:69) at java.base/jdk.internal.platform.CgroupUtil.lambda$readStringValue$1(CgroupUtil.java:66) at java.base/java.security.AccessController.doPrivileged(AccessController.java:554) at java.base/jdk.internal.platform.CgroupUtil.readStringValue(CgroupUtil.java:68) at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:65) at java.base/jdk.internal.platform.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:124) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getLongValue(CgroupV1Subsystem.java:272) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getHierarchical(CgroupV1Subsystem.java:218) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.setPath(CgroupV1Subsystem.java:201) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.setSubSystemControllerPath(CgroupV1Subsystem.java:173) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.lambda$initSubSystem$5(CgroupV1Subsystem.java:113) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.initSubSystem(CgroupV1Subsystem.java:113) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.(CgroupV1Subsystem.java:47) ... 33 more Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:65) at java.base/jdk.internal.platform.Container.metrics(Container.java:43) at jdk.management/com.sun.management.internal.OperatingSystemImpl.(OperatingSystemImpl.java:48) at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(PlatformMBeanProviderImpl.java:279) at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(PlatformMBeanProviderImpl.java:198) at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(PlatformMBeanProvider.java:195) at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:686) at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(ManagementFactory.java:388) at org.elasticsearch.tools.launchers.DefaultSystemMemoryInfo.(DefaultSystemMemoryInfo.java:29) at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:125) at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:61) ... 10 more Caused by: java.lang.ExceptionInInitializerError at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:107) at java.base/jdk.internal.platform.CgroupMetrics.getInstance(CgroupMetrics.java:167) ... 15 more Caused by: java.lang.NullPointerException at java.base/java.util.Objects.requireNonNull(Objects.java:208) at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:260) at java.base/java.nio.file.Path.of(Path.java:147) at java.base/java.nio.file.Paths.get(Paths.java:69) at java.base/jdk.internal.platform.CgroupUtil.lambda$readStringValue$1(CgroupUtil.java:66) at java.base/java.security.AccessController.doPrivileged(AccessController.java:554) at java.base/jdk.internal.platform.CgroupUtil.readStringValue(CgroupUtil.java:68) at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:65) at java.base/jdk.internal.platform.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:124) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getLongValue(CgroupV1Subsystem.java:272) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getHierarchical(CgroupV1Subsystem.java:218) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.setPath(CgroupV1Subsystem.java:201) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.setSubSystemControllerPath(CgroupV1Subsystem.java:173) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.lambda$initSubSystem$5(CgroupV1Subsystem.java:113) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.initSubSystem(CgroupV1Subsystem.java:113) at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.(CgroupV1Subsystem.java:47) ``` Note that Elastic operator is working fine. Can anyone tell me why the ES master fails? ---
**Author:** ![keshara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keshara/32/63979_2.png) [@keshara](https://discuss.elastic.co/u/keshara)\ **Post date:** [September 30, 2021, 3:14pm UTC](https://discuss.elastic.co/t/eck-deployment-fails-on-container-runtime-containerd/285488/2 "2021-09-30T15:14:54Z")
[Solved] Finally I was able to find the root cause and fix it. The subjected error was due to containerD starting on a linux system without enabling SystemdCgroup. That means, CentOS8 at this time of writing won't initialized containerD with SystemdCgroup awareness. To enable it; edit `/etc/containerd/config.toml` and insert a line `SystemdCgroup = true` after following line ```auto [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options\] ``` finally it should looks like ```auto [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options\] SystemdCgroup = ture ``` Note: **Indent** is required. WIth this setting applied, you can restart the service `systemctl restart containerd` and may be reboot the whole Kubernetes cluster. Once your cluster back in running, you can deploy the ECK without a problem. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 28, 2021, 3:15pm UTC](https://discuss.elastic.co/t/eck-deployment-fails-on-container-runtime-containerd/285488/3 "2021-10-28T15:15:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.