Custom Jenkins slave in Kubernetes doesn't have install apps

I have a Jenkinsfile setup in an internal project that uses a custom jenkins/slave build where I add nvm and Node 12 to perform the build. The custom image is in an internal Harbor instance I have running within my Kubernetes cluster. Everything seems to start fine, until it starts performing the pipeline steps. All the folders/files/applications I installed as part of my custom jenkins/slave is missing.

The Dockerfile for my jenkins/slave looks like this:

FROM jenkins/slave

MAINTAINER xxx

USER root

RUN apt-get update && apt-get install -y build-essential libssl-dev curl apt-transport-https ca-certificates software-properties-common

RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
RUN apt update && apt install docker-ce-cli

RUN curl -sL https://deb.nodesource.com/setup_12.x | bash -
RUN apt-get install -y nodejs

RUN mkdir /version_1.2.0

USER jenkins
RUN which node
RUN which npm
RUN node --version
RUN npm --version
RUN echo $PATH

And my Jenkinsfile:

pipeline {
  agent {
    kubernetes {
      label 'jenkins-slave-build'
      yaml """
kind: Pod
metadata:
  name: jenkins-slave-build
spec:
  containers:
  - name: jenkins-slave
    image: harbor.mydomain.com/library/custom/jenkinsslave:latest
    imagePullPolicy: Always
    tty: true
  restartPolicy: Never
"""
    }
  }
  stages {
    stage('Test') {
      steps {
        echo 'Testing..'
        sh 'ls -lart /'
        sh 'npm --version'
        sh 'node --version'
      }
    }
    ...
  }
}

When I run the pipeline I can see it pull the image from Harbor, it prints what looks like a correct YAML for the pod in the console output for the build, and I can see the pod spin up in Kubernetes if I do a kubectl get pods --watch. In my Jenkinsfile I try to do an ls at root, which should contain a folder version_1.2 but that folder is missing. Also I tried doing npm --version and node --version, but those fail saying the apps are not installed. My current thinking is that the pipeline is running on the Jenkins master, not the slave pod. I also tried using just the node:12.16.1 docker image as well, but that had the exact same results.