ECE Control planes node all containers down

[Icaka (Hristo Stoychev)]

I've been playing around and testing ECE (2.11.0) for one of our customers, so I have to use minimal footprint: 1 x node (node-1) for osting ECE COntrol plane (Coordinator, Director, Cloud UI) and 1 x node (node-2) for allocator and deployments. After restarting docker.service (dockerd) all containers on node-1 are gone. Surely all data is still there /mnt/data/docker and /mnt/data/elastic with volumes, services, logs and etc. However my ECE cluster is down and I don't know if I can bring it back and start those containers once again. That's my first question, second one is more important for me because, I would like to save and export all my custom Dashboards from Kibana (as I understood they should be located in elasticsearch index) that is located as container on node-2 which is allocator node and all containers are still running. How I can export my dashboards from ES container or to join them in new ECE cluster and use Kibana GUI export feature. Thanks in advance!

Regards,
Hristo.

[Icaka (Hristo Stoychev)]

After some troubleshooting I found and resolve my issue. I was going to expose Docker API and modified (override) daemon config file which cause the problem. After turn back all default settings to docker systemd config file and delete override files all went back up.
For the reference, /etc/systemd/system/docker.service.d/docker.conf (on RHEL family) should look like this:

[Unit]
Description=Docker Service
After=multi-user.target

[Service]
Environment="DOCKER_OPTS=-H unix:///run/docker.sock -g /mnt/data/docker --storage-driver=overlay2 --bip=172.17.42.1/16 --raw-logs --icc=false"
ExecStart=
ExecStart=/usr/bin/dockerd $DOCKER_OPTS
Restart=on-failure
RestartSec=1s
TimeoutSec=20

However, it's still unclear for me how docker locate and bring all containers back up to the status they were.
Regards,
Icaka.

[warkolm (Mark Walkom)]

Thanks for sharing this solution!