Docker/Podman in Docker fails as a GitLab CI job

Context

I am running my own GitLab instance with an extra server for CI. In case it matters my host for the gitlab runner has CentOS 8 and the gitlab runner is version 14.4.0 and the gitlab instance is running 14.4.1. I am currently using a shell executor on the CI server but want to switch to a docker executor.

Gitlab Runner Config

My /etc/gitlab-runner/config.toml looks like this:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "the old shell executor"
  url = "https://XXXXXXXXXXXXXXXXXXXXXX/"
  token = "XXXXXXXXXXXXXXXXXXXX"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

[[runners]]
  name = "the new docker executor"
  url = "https://XXXXXXXXXXXXXXXXXXXXXX/"
  token = "XXXXXXXXXXXXXXXXXXXX"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker.io/centos:7"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    network_mode = "host"
    shm_size = 0

The shell executor is tagged shell and the docker executor is tagges docker so I can decide where to run my build from the gitlab-ci.yml file.

gitlab-ci.yml

.template:
  before_script:
    - cat /proc/sys/user/max_user_namespaces
  script:
    - $ENGINE login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - $ENGINE build -f container/php-71.dockerfile -t "$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$TAG:$CI_PIPELINE_ID" .
    - $ENGINE push "$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$TAG:$CI_PIPELINE_ID"

build-with-docker-in-shell-executor:
  extends: .template
  variables:
    ENGINE: docker
    TAG: build-with-docker
  tags: [shell]
build-with-docker-in-docker-executor:
  extends: .template
  image: docker:dind
  variables:
    ENGINE: docker
    TAG: build-with-docker-in-docker
  tags: [docker]
build-with-podman-in-docker-executor:
  extends: .template
  image: quay.io/podman/stable
  variables:
    ENGINE: podman
    TAG: build-with-podman-in-docker
  tags: [docker]

My container/php-71.dockerfile is trivial, it uses centos:7 and executes a bunch of yum install commands:

FROM docker.io/centos:7
RUN yum install -y epel-release centos-release-scl
RUN yum install -y rh-php71

The first job fails because my gitlab runner user does not have access to the docker sockt:

Got permission denied while trying to connect to the Docker
daemon socket at unix:///var/run/docker.sock: Post
"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/auth": dial unix
/var/run/docker.sock: connect: permission denied

I think that is ok (or is it nowdays advisable to grant access to docker for non root users?). So this job exists just for reference.

The docker in docker job fails because yum does not have network access it seems:

Step 1/3 : FROM docker.io/centos:7
 ---> eeb6ee3f44bd
Step 2/3 : RUN yum install -y epel-release centos-release-scl
 ---> Running in 8e41cd05bcc1
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Resolving timed out after 30548 milliseconds')
 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:
     1. Contact the upstream for the repository and get them to fix the problem.
     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).
     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...
     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:
            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>
     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:
            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot find a valid baseurl for repo: base/7/x86_64
The command '/bin/sh -c yum install -y epel-release centos-release-scl' returned a non-zero code: 1

And the podman in docker is the strangest errors of them all, it already fails at the podman login command:

time="2021-11-01T13:34:31Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
cannot clone: Operation not permitted
Error: cannot re-exec process

Question

I would like to use podman in docker or docker in docker to build my images. What steps can I take to debug and fix this?