I am trying to set up a Nexus repository for use by a Docker Swarm. However, I am seeing some weird behaviour and I am not sure if it is something I did or if it is a bug because it appears to be present since 3.12 (that's the first version that had the S3 blob store so that was the first time I used Nexus)
The behaviour I am seeing is docker pull
works but when I do a docker swarm deploy
it says the image could not be accessed. I don't have the issue with the registry:2
image.
Here's the output from docker pull
which shows it working successfully. It also means that I have authenticated correctly.
$ sudo docker pull repo.devhaus.com/mylj-api:latest
latest: Pulling from mylj-api
Digest: sha256:e6ca3333cbe6d8c1761b2af2548610b068e1a268dac437598fb76a5b4102f9eb
Status: Downloaded newer image for repo.devhaus.com/mylj-api:latest
The docker-compose.yml
file used for deployment is
version: '3.4'
services:
apidoc:
image: repo.devhaus.com/mylj-api
However, when I attempted to deploy I got
$ sudo docker stack deploy apidoc -c apidoc.yml --with-registry-auth --prune
Updating service apidoc_apidoc (id: odj2sgw8zugthjfk4jom1erms)
image repo.devhaus.com/mylj-api:latest could not be accessed on a registry to record
its digest. Each node will access repo.devhaus.com/mylj-api:latest independently,
possibly leading to different nodes running different
versions of the image.
I have also attempted to give the users nx-admin
rights which they really shouldn't because all they need to do is read from the repository, not write.
In /var/log/messages
the following appears on docker stack deploy
but not in docker pull
Oct 17 01:25:59 ip-10-70-1-11 dockerd: time="2018-10-17T01:25:59.853065022Z" level=error msg="Handler for GET /v1.38/distribution/repo.devhaus.com/mylj-api:latest/json returned error: missing or empty Content-Length header"
I am thinking that docker stack deploy
may be doing a HEAD
request.
The following works:
curl --user XXXX:YYYY https://repo.devhaus.com/v2/mylj-api/manifests/latest
The following which is basically the HEAD
request equivalent gives a 404 :
curl -I --user XXXX:YYYY https://repo.devhaus.com/v2/mylj-api/manifests/latest does not and gives a 404
I also confirmed that it was not a proxy issue as I hit the exposed port directly and yielded the same results as above.
Apparently it is a bug in Nexus since 3.2.0 since March 2017 (so it's over a year old) https://issues.sonatype.org/browse/NEXUS-12684 I guess we need to find an alternative repository system that supports Docker.
Or as a workaround for the time being, I created a simple proxy server that allows improperly built servers to support a
HEAD
request. https://hub.docker.com/r/trajano/fakehead/