// ISSUE //
From my test VM, pull image from elastic docker registry failed, returning "x509: certificate signed by unknown authority"
# docker pull docker.elastic.co/elasticsearch/elasticsearch:7.14.0
Error response from daemon: Get https://docker.elastic.co/v2/: x509: certificate signed by unknown authority
// TROUBLESHOOTING //
(1) Verify the elastic docker registry is reachable via port 443:
# telnet docker.elastic.co 443
Trying 34.68.230.202...
Connected to docker.elastic.co.
Escape character is '^]'.
(2) Verify my VM's CA don't trust the certificate from elastic docker registry:
# curl https://docker.elastic.co:443
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
# curl --trace - https://docker.elastic.co:443
== Info: About to connect() to docker.elastic.co port 443 (#0)
== Info: Trying 34.68.230.202...
== Info: Connected to docker.elastic.co (34.68.230.202) port 443 (#0)
== Info: Initializing NSS with certpath: sql:/etc/pki/nssdb
== Info: CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
== Info: Server certificate:
== Info: subject: CN=docker.elastic.co
== Info: start date: Aug 11 08:49:30 2021 GMT
== Info: expire date: Nov 09 08:49:28 2021 GMT
== Info: common name: docker.elastic.co
== Info: issuer: <xxxxxx - masked>
== Info: NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
== Info: Peer's certificate issuer has been marked as not trusted by the user.
== Info: Closing connection 0
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
(3) Pull other image from Docker Hub successfully:
# docker pull ubuntu:14.04
14.04: Pulling from library/ubuntu
2e6e20c8e2e6: Pull complete
0551a797c01d: Pull complete
512123a864da: Pull complete
Digest: sha256:43cb19408de1e0ecf3ba5b5372ec98978963d6d0be42d0ad825e77a3bd16b5f7
Status: Downloaded newer image for ubuntu:14.04
docker.io/library/ubuntu:14.04
(4) Verify my VM CA trust the certificate from Docker Hub (docker.io):
# curl https://docker.io:443
# curl --trace - https://docker.io:443
== Info: About to connect() to docker.io port 443 (#0)
== Info: Trying 34.192.247.57...
== Info: Connected to docker.io (34.192.247.57) port 443 (#0)
== Info: Initializing NSS with certpath: sql:/etc/pki/nssdb
== Info: CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
== Info: SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
== Info: Server certificate:
== Info: subject: CN=*.docker.io
== Info: start date: Apr 25 00:00:00 2021 GMT
== Info: expire date: May 24 23:59:59 2022 GMT
== Info: common name: *.docker.io
== Info: issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
=> Send header, 73 bytes (0x49)
0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
0010: 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c User-Agent: curl
0020: 2f 37 2e 32 39 2e 30 0d 0a 48 6f 73 74 3a 20 64 /7.29.0..Host: d
0030: 6f 63 6b 65 72 2e 69 6f 0d 0a 41 63 63 65 70 74 ocker.io..Accept
0040: 3a 20 2a 2f 2a 0d 0a 0d 0a : */*....
<= Recv header, 32 bytes (0x20)
0000: 48 54 54 50 2f 31 2e 31 20 33 30 31 20 4d 6f 76 HTTP/1.1 301 Mov
0010: 65 64 20 50 65 72 6d 61 6e 65 6e 74 6c 79 0d 0a ed Permanently..
<= Recv header, 19 bytes (0x13)
0000: 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 content-length:
0010: 30 0d 0a 0..
<= Recv header, 35 bytes (0x23)
0000: 6c 6f 63 61 74 69 6f 6e 3a 20 68 74 74 70 73 3a location: https:
0010: 2f 2f 77 77 77 2e 64 6f 63 6b 65 72 2e 63 6f 6d //www.docker.com
0020: 2f 0d 0a /..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a ..
== Info: Connection #0 to host docker.io left intact
(5) Did some search and tried out several ways as per these documents, but issue persist:
https://docs.docker.com/engine/security/certificates/
https://docs.docker.com/registry/insecure/#/docker-still-complains-about-the-certificate-when-using-authentication
https://forums.docker.com/t/docker-private-registry-x509-certificate-signed-by-unknown-authority/21262
(6) From an external online Docker playground, is able to pull the elastic image, so this proves issue is not on elastic side.
$ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.14.0
7.14.0: Pulling from elasticsearch/elasticsearch
ddf49b9115d7: Pull complete
693712dd4782: Pull complete
e1843b23879c: Pull complete
43b0526f3562: Pull complete
395a8174c4c5: Pull complete
671bce37ebcd: Pull complete
7b0bb020b357: Pull complete
Digest: sha256:46815a329dfdf633cac9261e287c8440512db13554cbcb4fa89842dde58e74b2
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.14.0
docker.elastic.co/elasticsearch/elasticsearch:7.14.0
(7) Found out a post on this forum:
https://discuss.elastic.co/t/docker-elastic-6-unknown-authority/123863
As per Dimitrios Liappis: "As you probably know, by default, all docker * commands towards docker.elastic.co will use https and verify its authentication through the certificate of docker.elastic.co. The elastic docker registry will not accept non-https connections."
As per the PO's last update, "It was because of my network security configuration (not allowed to hit https with certificate signed by an unknown authority)."
// ROOT CAUSE //
I believe the issue I'm having is exactly the same root cause as the above post. I believe the network security doesn't trust unknown authority and block it.