Elastic-operator-0 CrashLoopBackOff

[SnakeByte (Ian)]

Following the instructions here, I only got as far as step 2 before running into trouble :
Deploy ECK in your Kubernetes cluster

elastic-operator-0 is left in a CrashLoogBackOff state, with these three log entries:

{"log.level":"error","@timestamp":"2021-03-19T01:16:23.369Z","log.logger":"manager","message":"Error setting GOMAXPROCS","service.version":"1.4.1+512d5431","service.type":"eck","ecs.version":"1.4.0","error":"invalid format for CGroupSubsys: \"12:freezer:/kubepods-burstable-podb913058b_6a87_453c_b589_bc8e84d29adb.slice:cri-containerd:6e4fa8c12a6d16cf94a3acb4b9edd5cc8314d85537a11115c8c9f40c55ca0796\"","error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.startOperator\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:362\ngithub.com/elastic/cloud-on-k8s/cmd/manager.doRun.func2\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:321"}

{"log.level":"error","@timestamp":"2021-03-19T01:16:23.369Z","log.logger":"manager","message":"Shutting down due to error","service.version":"1.4.1+512d5431","service.type":"eck","ecs.version":"1.4.0","error":"invalid format for CGroupSubsys: \"12:freezer:/kubepods-burstable-podb913058b_6a87_453c_b589_bc8e84d29adb.slice:cri-containerd:6e4fa8c12a6d16cf94a3acb4b9edd5cc8314d85537a11115c8c9f40c55ca0796\"","error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.doRun\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:329\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:850\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:895\nmain.main\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/main.go:30\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}

Error: invalid format for CGroupSubsys: "12:freezer:/kubepods-burstable-podb913058b_6a87_453c_b589_bc8e84d29adb.slice:cri-containerd:6e4fa8c12a6d16cf94a3acb4b9edd5cc8314d85537a11115c8c9f40c55ca0796"

From what I could find searching the internet, this is a golang thing, but I'm uncertain why there is trouble. The node is running Ubuntu 18.04 LTS as a guest vm within Vmware Workstation 12 Pro that's running on top of Windows 10 Pro (Intel CPU). The runtime is Docker 19.03.8, with the cgroupdriver set to systemd.

[sebgl (Sebastien Guilloux)]

@SnakeByte

invalid format for CGroupSubsys

This seems related: Invalid format for CGroupSubsys · Issue #4083 · elastic/cloud-on-k8s · GitHub.

[SnakeByte (Ian)]

Thanks sebgl -- apparently I was using containerd for the runtime after-all -- I thought I blew that node away. Your link eventually got me to the solution:

version = 2
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
   [plugins."io.containerd.grpc.v1.cri".containerd]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v1"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

for /etc/containerd/config.toml

Oddly, the instructions for creating the containerd configuration file here (Container runtimes | Kubernetes) made a configuration file that I couldn't just edit and inline the change above -- I had to completely replace the file with just the contents above for this to start working.