Kokos Bot

Asked: 2021-02-11 18:47:20 +0800 CST2021-02-11 18:47:20 +0800 CST 2021-02-11 18:47:20 +0800 CST

Kaniko (GitLab CI, K8s) can not push to Nexus Repository Manager: UNAUTHORIZED: access to the requested resource is not authorized

my GitLab CI job build a example container image with Kaniko with the GitLab runner on K8s. All services are in my LAN behind a static IP. The build process pulls an ArchLinux image from my first Nexus registry service under secure domain A. Then I try do push it to the same Nexus registry into a second snapshot repo under a second domain B. Therefor I use the same Nexus user. If the job try to push, it results in the following error:

$ /kaniko/executor  --context $CI_PROJECT_DIR  --dockerfile $CI_PROJECT_DIR/Dockerfile  --destination ${SNAPSHOT_CONTAINER_REGISTRY}/${IMAGE_NAME}

error checking push permissions -- make sure you entered the correct tag name, 
and that you are authenticated correctly, and try again: 
checking push permission for "B/example-app:dev-proto-kokosbot": 
POST https://B/v2/example-app/blobs/uploads/: 
UNAUTHORIZED: access to the requested resource is not authorized

Even if I grant the cicd user admin rights I can not push over kaniko. With the docker CLI pushing is no problem and I can reach the URL of the error in the browser and it works...

My CI config:

stages:
  - build

variables:
  COMMIT_SHORT_SHA: ${CI_COMMIT_SHA} at ${CI_COMMIT_BRANCH}
  IMAGE_NAME: example-app:${CI_COMMIT_BRANCH}
  
build-job:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo [INFO] Start building container ${IMAGE_NAME}
    - echo [INFO] Target container registry ${SNAPSHOT_CONTAINER_REGISTRY}
    - mkdir -p /kaniko/.docker
    - |
      echo "{\"auths\":{
        \"${CONTAINER_REGISTRY}\":{
          \"auth\":\"${AUTH_REGISTRY}\"
          }
        },
        \"${SNAPSHOT_CONTAINER_REGISTRY}\":{
          \"auth\":\"${AUTH_REGISTRY}\"
          }
        }        
      }" > /kaniko/.docker/config.json
    - less /kaniko/.docker/config.json
    - >
      /kaniko/executor 
      --context $CI_PROJECT_DIR 
      --dockerfile $CI_PROJECT_DIR/Dockerfile 
      --destination ${SNAPSHOT_CONTAINER_REGISTRY}/${IMAGE_NAME}
      --no-push
    - >
      /kaniko/executor 
      --context $CI_PROJECT_DIR 
      --dockerfile $CI_PROJECT_DIR/Dockerfile 
      --destination ${SNAPSHOT_CONTAINER_REGISTRY}/${IMAGE_NAME}

I have no idea where to start to understand the problem, does anyone have a clue?

Greetings, Kokos Bot.

GitLab CE
Nexus Repository Manager: Free editon

EDIT: Do I understand it correctly that simply the user is not recognized? The user is called CICD and not *UNKOWN.

Nexus log:

nexus_1  | 2021-02-11 06:48:20,719+0000 INFO  [qtp874955081-232] *UNKNOWN org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Enabling session validation scheduler...
nexus_1  | 2021-02-11 06:48:20,739+0000 INFO  [qtp874955081-232] *UNKNOWN org.sonatype.nexus.internal.security.anonymous.AnonymousManagerImpl - Loaded configuration: OrientAnonymousConfiguration{enabled=false, userId='anonymous', realmName='NexusAuthorizingRealm'}
nexus_1  | 2021-02-11 06:48:20,741+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Servicing: GET /repository/dio-container-snapshot/v2/ (https://snapshot.container.ABC.ABC/repository/dio-container-snapshot/v2/)
nexus_1  | 2021-02-11 06:48:20,742+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Parsed path: RepositoryPath{repositoryName='dio-container-snapshot', remainingPath='/v2/'}
nexus_1  | 2021-02-11 06:48:20,742+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Looking for repository: dio-container-snapshot
nexus_1  | 2021-02-11 06:48:20,743+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Repository: RepositoryImpl$$EnhancerByGuice$$765e645e{type=hosted, format=docker, name='dio-container-snapshot'}
nexus_1  | 2021-02-11 06:48:20,743+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Dispatching to view facet: org.sonatype.nexus.repository.view.ConfigurableViewFacet$$EnhancerByGuice$$d744b4fe@6a0c93a4
nexus_1  | 2021-02-11 06:48:20,745+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.HttpResponseSenderSelector - Looking for HTTP response sender: docker
nexus_1  | 2021-02-11 06:48:20,747+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Request: Request{action='GET', path='/v2/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='null', size=-1}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,748+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@2503258 AND LiteralMatcher{literal='/v2/', ignoreCase=true}
nexus_1  | 2021-02-11 06:48:20,748+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.LiteralMatcher - Matching: /v2/=/v2/ ignore-case: true
nexus_1  | 2021-02-11 06:48:20,749+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Starting: Route{matcher=AndMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.ActionMatcher@2503258, LiteralMatcher{literal='/v2/', ignoreCase=true}]}, handlers=[org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347, org.sonatype.nexus.repository.security.SecurityHandler@63226ff1, org.sonatype.nexus.repository.view.handlers.HandlerContributor@60e2348c, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure2@57297d]}
nexus_1  | 2021-02-11 06:48:20,749+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d
nexus_1  | 2021-02-11 06:48:20,749+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9
nexus_1  | 2021-02-11 06:48:20,752+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347
nexus_1  | 2021-02-11 06:48:20,752+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.security.SecurityHandler@63226ff1
nexus_1  | 2021-02-11 06:48:20,800+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,801+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Request: Request{action='GET', path='/v2/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='null', size=-1}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,801+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender - Sending response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,806+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Service completed
nexus_1  | 2021-02-11 06:48:20,820+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Servicing: POST /repository/dio-container-snapshot/v2/example-app/blobs/uploads/ (https://snapshot.container.ABC.ABC/repository/dio-container-snapshot/v2/example-app/blobs/uploads/)
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Parsed path: RepositoryPath{repositoryName='dio-container-snapshot', remainingPath='/v2/example-app/blobs/uploads/'}
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Looking for repository: dio-container-snapshot
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Repository: RepositoryImpl$$EnhancerByGuice$$765e645e{type=hosted, format=docker, name='dio-container-snapshot'}
nexus_1  | 2021-02-11 06:48:20,821+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Dispatching to view facet: org.sonatype.nexus.repository.view.ConfigurableViewFacet$$EnhancerByGuice$$d744b4fe@6a0c93a4
nexus_1  | 2021-02-11 06:48:20,822+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.HttpResponseSenderSelector - Looking for HTTP response sender: docker
nexus_1  | 2021-02-11 06:48:20,822+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Request: Request{action='POST', path='/v2/example-app/blobs/uploads/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='application/json', size=0}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,822+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@2503258 AND LiteralMatcher{literal='/v2/', ignoreCase=true}
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@2fb965f0 AND OrMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@6264781a, org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@3679c546]}
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@291c9924 AND OrMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@4deeef12, org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@c0bea1d]}
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@74a04d2f AND org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@39b83f26
nexus_1  | 2021-02-11 06:48:20,823+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.logic.AndMatcher - Matching: org.sonatype.nexus.repository.view.matchers.ActionMatcher@314d17c AND org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@501c8801
nexus_1  | 2021-02-11 06:48:20,824+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.matchers.token.TokenMatcher - Matching: /v2/example-app/blobs/uploads/~=TokenParser{pattern=\Q/v2/\E(?<name>.+)\Q/blobs/uploads/\E, variables=[var(name,.+)]}
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Starting: Route{matcher=AndMatcher{matchers=[org.sonatype.nexus.repository.view.matchers.ActionMatcher@314d17c, org.sonatype.nexus.repository.view.matchers.token.TokenMatcher@501c8801]}, handlers=[org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347, org.sonatype.nexus.repository.security.SecurityHandler@63226ff1, org.sonatype.nexus.repository.view.handlers.HandlerContributor@60e2348c, org.sonatype.nexus.repository.storage.UnitOfWorkHandler@7ba0816b, org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure3@4f48429e]}
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.view.handlers.TimingHandler@38a76c9d
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1@64b09cb9
nexus_1  | 2021-02-11 06:48:20,825+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18@60e34347
nexus_1  | 2021-02-11 06:48:20,826+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Context - Proceeding: org.sonatype.nexus.repository.security.SecurityHandler@63226ff1
nexus_1  | 2021-02-11 06:48:20,827+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.view.Router - Response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,827+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Request: Request{action='POST', path='/v2/example-app/blobs/uploads/', parameters={}, payload=HttpRequestPayloadAdapter{contentType='application/json', size=0}, multipart=false}
nexus_1  | 2021-02-11 06:48:20,828+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender - Sending response: Response{status=Status{successful=false, code=401, message='null'}, payload=StringPayload{size=113, charset=UTF-8, contentType='application/json'}}
nexus_1  | 2021-02-11 06:48:20,828+0000 DEBUG [qtp874955081-232] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Service completed

1 Answers

Voted

Best Answer

Kokos Bot
2021-02-12T00:45:59+08:002021-02-12T00:45:59+08:00
I have found the problem. There is one faulty bracket in the Docker config. Unfortunately with the environment variables there are many brackets. Therefore this is not easy noticeable.

The more astonishing it is that Kaniko, with a verbosity of tracing, does not spit out an error...
1

Kaniko (GitLab CI, K8s) can not push to Nexus Repository Manager: UNAUTHORIZED: access to the requested resource is not authorized

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?