Elastic-operator-0 CrashLoopBackOff

Following the instructions here, I only got as far as step 2 before running into trouble :slight_smile: :
Deploy ECK in your Kubernetes cluster

elastic-operator-0 is left in a CrashLoogBackOff state, with these three log entries:

{"log.level":"error","@timestamp":"2021-03-19T01:16:23.369Z","log.logger":"manager","message":"Error setting GOMAXPROCS","service.version":"1.4.1+512d5431","service.type":"eck","ecs.version":"1.4.0","error":"invalid format for CGroupSubsys: \"12:freezer:/kubepods-burstable-podb913058b_6a87_453c_b589_bc8e84d29adb.slice:cri-containerd:6e4fa8c12a6d16cf94a3acb4b9edd5cc8314d85537a11115c8c9f40c55ca0796\"","error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.startOperator\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:362\ngithub.com/elastic/cloud-on-k8s/cmd/manager.doRun.func2\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:321"}

{"log.level":"error","@timestamp":"2021-03-19T01:16:23.369Z","log.logger":"manager","message":"Shutting down due to error","service.version":"1.4.1+512d5431","service.type":"eck","ecs.version":"1.4.0","error":"invalid format for CGroupSubsys: \"12:freezer:/kubepods-burstable-podb913058b_6a87_453c_b589_bc8e84d29adb.slice:cri-containerd:6e4fa8c12a6d16cf94a3acb4b9edd5cc8314d85537a11115c8c9f40c55ca0796\"","error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.doRun\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:329\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:850\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:895\nmain.main\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/main.go:30\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}

Error: invalid format for CGroupSubsys: "12:freezer:/kubepods-burstable-podb913058b_6a87_453c_b589_bc8e84d29adb.slice:cri-containerd:6e4fa8c12a6d16cf94a3acb4b9edd5cc8314d85537a11115c8c9f40c55ca0796"

From what I could find searching the internet, this is a golang thing, but I'm uncertain why there is trouble. The node is running Ubuntu 18.04 LTS as a guest vm within Vmware Workstation 12 Pro that's running on top of Windows 10 Pro (Intel CPU). The runtime is Docker 19.03.8, with the cgroupdriver set to systemd.

This seems related: Invalid format for CGroupSubsys · Issue #4083 · elastic/cloud-on-k8s · GitHub.

1 Like

Thanks sebgl -- apparently I was using containerd for the runtime after-all -- I thought I blew that node away. Your link eventually got me to the solution:

version = 2
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
   [plugins."io.containerd.grpc.v1.cri".containerd]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v1"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

for /etc/containerd/config.toml

Oddly, the instructions for creating the containerd configuration file here (Container runtimes | Kubernetes) made a configuration file that I couldn't just edit and inline the change above -- I had to completely replace the file with just the contents above for this to start working.