I have GitLab installed in Kubernetes with their Helm chart.
I migrated my old Gitlab deployment from one cluster to another with the following steps:
- Scale down all pods in old cluster
- Apply values.yml with helm to new cluster (to create PVCs)
- Scale down all pods in new cluster
- Change DNS records, HAProxy, etc
- Manually rsync data from old PVCs to new PVCs (minio, gitaly, redis, postgres, prometheus)
- Run helm upgrade to bring deployments back online in new cluster
After all that the deployment for the most part works fine. Able to login and use git.
But the runner is failing to register, so I can't run any CI. Looking at the gitlab-gitlab-runner pod, I see the message below repeated over and over:
Registration attempt 30 of 30
Runtime platform arch=amd64 os=linux pid=691 revision=3b6f852e version=14.0.0
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
ERROR: Registering runner... failed runner=y6ixJoR1 status=500 Internal Server Error
PANIC: Failed to register the runner. You may be having network problems.
As you can see, it's failing to register the runner. Trying to go to /admin/runners gives me a 500 error.
Where can I see more information as to why I am getting this 500 error?
It looks like you have restored data and configuration from the previous cluster, however you are missing the secrets on the new cluster. Here you can find a procedure, what should you do, when the secrets are lost. You can also find there:
See also similar problems:
A: Assure you've restored
/etc/gitlab/gitlab-secrets.jsonfile.A: It's in
/var/log/gitlab/gitlab-rails/production.log