Why does docker pull
from a private Docker registry using an nginx SSL reverse proxy work on registry host, but not remote host?
Successful docker pull
From registry host:
admin@vps58622:~/docker-registry$ docker pull www.opessolutions.com:5043/busybox
Using default tag: latest
latest: Pulling from busybox
Digest: sha256:6757d4b17cd75742fc3b1fc1a8d02b45b637f2ac913ee9669f5c2aed0c9b26ba
Status: Image is up to date for www.opessolutions.com:5043/busybox:latest
Docker logs:
nginx_1 | 158.69.212.126 - - [25/May/2016:16:41:54 +0000] "GET /v2/ HTTP/1.1" 401 195 "-" "docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.0-22-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \x5C(linux\x5C))" "-"
registry_1 | time="2016-05-25T16:41:54Z" level=info msg="response completed" go.version=go1.6.1 http.request.host="www.opessolutions.com:5043" http.request.id=4302d195-a4b4-4dfb-93b3-802ab6cacf28 http.request.method=GET http.request.remoteaddr=158.69.212.126 http.request.uri="/v2/busybox/manifests/latest" http.request.useragent="docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.0-22-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \\(linux\\))" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=6.762288ms http.response.status=200 http.response.written=711 instance.id=701faf44-22ad-4386-9305-46a0167b95c8 version=v2.4.0
registry_1 | 172.17.0.3 - - [25/May/2016:16:41:54 +0000] "GET /v2/busybox/manifests/latest HTTP/1.0" 200 711 "" "docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.0-22-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \\(linux\\))"
nginx_1 | 158.69.212.126 - derek [25/May/2016:16:41:54 +0000] "GET /v2/busybox/manifests/latest HTTP/1.1" 200 711 "-" "docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.0-22-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \x5C(linux\x5C))" "-"
Failed docker pull
from remote host:
derek@derek-lubuntu:~/Projects/docker-library$ docker pull www.opessolutions.com:5043/busybox
Using default tag: latest
Pulling repository www.opessolutions.com:5043/busybox
Error: image busybox not found
Docker logs:
nginx_1 | 66.171.168.90 - - [25/May/2016:16:42:05 +0000] "GET /v2/ HTTP/1.1" 401 195 "-" "docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.0-22-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \x5C(linux\x5C))" "-"
nginx_1 | 2016/05/25 16:42:06 [error] 7#7: *5 open() "/etc/nginx/html/v1/_ping" failed (2: No such file or directory), client: 66.171.168.90, server: www.opessolutions.com, request: "GET /v1/_ping HTTP/1.1", host: "www.opessolutions.com:5043"
nginx_1 | 66.171.168.90 - - [25/May/2016:16:42:06 +0000] "GET /v1/_ping HTTP/1.1" 404 169 "-" "docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.0-22-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \x5C(linux\x5C))" "-"
nginx_1 | 2016/05/25 16:42:06 [error] 7#7: *6 open() "/etc/nginx/html/v1/repositories/busybox/images" failed (2: No such file or directory), client: 66.171.168.90, server: www.opessolutions.com, request: "GET /v1/repositories/busybox/images HTTP/1.1", host: "www.opessolutions.com:5043"
nginx_1 | 66.171.168.90 - - [25/May/2016:16:42:06 +0000] "GET /v1/repositories/busybox/images HTTP/1.1" 404 169 "-" "docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.0-22-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \x5C(linux\x5C))" "-"
Notice that in the failed attempt, nginx doesn't forward the request to Docker Registry. It also seems to translate the Docker Registry API v2 request into a v1 request.
nginx/registry.conf
upstream docker-registry {
server registry:5000;
}
## Set a variable to help us decide if we need to add the
## 'Docker-Distribution-Api-Version' header.
## The registry always sets this header.
## In the case of nginx performing auth, the header will be unset
## since nginx is auth-ing before proxying.
map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
'registry/2.0' '';
default registry/2.0;
}
server {
listen 443 ssl;
server_name www.opessolutions.com;
# SSL
ssl_certificate /etc/nginx/conf.d/domain.crt;
ssl_certificate_key /etc/nginx/conf.d/domain.key;
# Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
# To add basic authentication to v2 use auth_basic setting.
auth_basic "Registry realm";
auth_basic_user_file /etc/nginx/conf.d/registry.htpasswd;
## If is empty, the header will not be added.
## See the map directive above where this variable is defined.
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
proxy_pass http://docker-registry;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
docker-compose.yml
nginx:
image: nginx:1.9
ports:
- 5043:443
links:
- registry:registry
volumes:
- ./nginx:/etc/nginx/conf.d:ro
registry:
image: registry:2
ports:
- 127.0.0.1:5000:5000
environment:
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
volumes:
- ./data:/data
Turns out that the remote
docker pull
didn't work because I had neglected to first login to the registry from the remote host, but had logged in to the registry from the registry host. As soon as I did login to the registry from the remote host, the remote pull worked:Unfortunately, the error message that
docker pull
reported and the error messages in the nginx logs, didn't help much to troubleshoot this problem.