ECE not recognizing docker config.json

[avarshney (Abhinav)]

We have config.json file which has docker registry auth details of our private artifactory registry. However, when we create a new deployment, ECE is not able to find the auth details and tries to connect to the registry as “anonymous” (authentication is required for private registry). We are trying to avoid situation where we need to do a separate docker pull before ECE tries (ECE should be able to find the auth for docker registry – bolded red below).

Currently 7.5 docker images dont exist on the EC2. We are expecting ECE to download them automatically if they dont exist already on the EC2 host. Is this expectation correct?

Is there a way to confirm ECE is able to read the ~/.docker/config.json or does it look for the auth for registry in a different location? Also, is there a specific format that ECE expects the file to be in? Below is the format we have currently.

Docker auth file:
[elastic@ip-XX-XXX-XX-X logs] ls -tlr ~/.docker/config.json -rw-r--r--. 1 elastic elastic 183 Dec 13 13:12 /home/elastic/.docker/config.json [elastic@ip-XX-XXX-XX-X logs]
{
"auths": {
"artifactory-dev.com/ct-cdo-dockerece-remote": {
"auth": "XXXXXXXXXXXXXXXXXXXXXXXXX"
}
}
}

Allocator log:

[2019-12-13 18:57:45,118][INFO ][no.found.runner.docker.ContainerInspector] Container not found. {"ec_container_kind":"elasticsearch","ec_container_group":"e5c3d0e524d04888aff7bc6f63f1ec2d","ec_container_name":"instance-0000000
002"}
[2019-12-13 18:57:47,259][WARN ][no.found.docker.DockerContainerManager] Unable to find auth for registry [artifactory-dev.com/ct-cdo-dockerece-remote] in auths. Have [List()], trying without auth. {}
[2019-12-13 18:57:47,259][INFO ][no.found.docker.DockerContainerManager] Pulling image [artifactory-dev.mlp.com/ct-cdo-dockerece-remote/cloud-assets/elasticsearch:7.5.0-0] {}
[2019-12-13 18:57:48,070][ERROR][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainerManager] Unexpected error during allocation {"ec_container_kind":"elasticsearch","ec_container_group":"e5c3d0e524d04888aff7bc
6f63f1ec2d","ec_container_name":"instance-0000000002"}
no.found.docker.DockerApiException: Unable to pull image [artifactory-dev.com/ct-cdo-dockerece-remote/cloud-assets/elasticsearch:7.5.0-0]

[Alex_Piggott]

I'll look into this more on Monday: we do have the capability to specify auths, but the API is somewhat hidden in ECE (we used to use it internally but don't any more).

If possible, I'd remove the auths from the docker registry (none of the images are actually sensitive) - if that's not possible for policy/pragmatic/etc reasons, I'd complain to your support rep and we can give you a script to use the internal API to configure it.

[avarshney (Abhinav)]

Hi Alex, Thanks. Unfortunately, we cant turn off authentication on the registry due to internal security concerns).
From the logs, it looks like ECE tries to look for the auth details from "somewhere". Is that what you referred to as "internal API"? Shouldnt ECE look at the standard docker auth file for the credentials?

[avarshney (Abhinav)]

Just to add, received a workaround which works. It's not ideal though since it requires us to tinker with the containers. Is there any alternate better solution?
"On each allocator (inside frc-allocators-allocator docker container), create /home/elastic/.docker/config.json and copy contents from the main host's file /home/elastic/.docker/config.json. This will allow the Allocator to successfully authenticate to the Artifactory registry and download required images."
Commands:

1. docker ps | grep -i allocators-allocator
   2e8d2e7af2f5 artifactory-dev.com/ct-cdo-dockerece-remote/cloud-enterprise/elastic-cloud-enterprise:2.2.3 "/sbin/my_init" 5 days ago Up 4 hours frc-allocators-allocator
2. docker exec -it 2e8d2e7af2f5 /bin/bash
3. mkdir -p /home/elastic/.docker
4. vi config.json
   a. Paste the contents of authentication in this file from /home/elastic/.docker/config.json residing on the underlying host.

[Alex_Piggott]

@avarshney

Is there any alternate better solution?

I think working with support to use the private API (as a one off) is probably preferred - the above is a fine interim while waiting to sort that out (btw you can just do docker exec -it frc-allocators-allocator bash for step 1+2)

[avarshney (Abhinav)]

Hi Alex,
Would you know what that private API is? or is there any docs related to it that i can use?

[Alex_Piggott]

@avarshney - we are working towards opening a safe and easy-to-understand version of the API question, but currently it's an internal API that is a) very coupled to the internal implementation of our infrastructure, b) very easy to use to bring down your platform!

As a result our support engineers work with ECE users to build them custom scripts based on the API (and ensure that people with a deep understanding of that API are on hand to help) - so contacting support is the right way to go for that

Alex