containerd 1.4.9 Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

Had the same error today while upgrade kubelet on the worker nodes. Issue was within the default configuration. Note that containerd will run fine without any config. In my case I just wanted to enable systemd_cgroup.

ctr plugin ls showed that cri plugin was in error state with default configuration

Just a blank config with systemd_cgroup fixed issue for me:

cat > /etc/containerd/config.toml <<EOF
[plugins."io.containerd.grpc.v1.cri"]
  systemd_cgroup = true
EOF
systemctl restart containerd

Background Context About the error:
From gitlab.cncf.ci/containerd crictl.md docs

"This could be that you are using an incorrect containerd configuration (maybe from a Docker install). You will need to update your containerd configuration to the containerd instance that you are running."

• I myself had installed docker, then yum installed crictl to investigate command syntax differences and ran into this.
• The resolution command posted in the linked doc only works if run as root, so here's a more generic version.

# Backup old containerd config (optional)
sudo mv /etc/containerd/config.toml /etc/containerd/config.bak

# Regenerate containerd config
sudo containerd config default | sudo tee /etc/containerd/config.toml

# Restart containerd
sudo systemctl restart containerd

# The above got it to work for me; but with some warnings 
# and ignorable errors that looked this this: 
sudo crictl ps
# WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead.
# ERRO[0002] connect endpoint 'unix:///var/run/dockershim.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
# WARN[0002] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead.
# ERRO[0004] connect endpoint 'unix:///var/run/dockershim.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
# CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
# ^-- The last line represents correct output, which is why 
# I say ignorable warnings/errors, even the post command 
# exit code seeable using 'echo $?' exit code shows success

# What cleaned up the errors for me was copy pasting the following
echo """
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
""" | sudo tee /etc/crictl.yaml

docker ps
# ^-- no more errors :)

# Note others may need to run one of these instead, based on their
# system's config, keep trying docker ps until one config works
echo """
runtime-endpoint: unix:///var/run/crio/crio.sock
image-endpoint: unix:///var/run/crio/crio.sock
""" | sudo tee /etc/crictl.yaml

echo """
runtime-endpoint: unix:///var/run/dockershim.sock
image-endpoint: unix:///var/run/dockershim.sock
""" | sudo tee /etc/crictl.yaml

This problem related to errors in CRI plugins. You can check the status of the CRI plugin

ctr plugin ls

In the past I got the same problem due to devmapper problem, since devmapper is configured as default CRI snapshotter, CRI got error as well.

TYPE                                  ID                       PLATFORMS      STATUS
io.containerd.snapshotter.v1          devmapper                linux/amd64    error
io.containerd.grpc.v1                 cri                      linux/amd64    error

The problem gone after I reconfigure the devmapper snapshotter.

Removing the config (/etc/containerd/config.toml) works as well, but the containerd runs with default configuration, which was not what I wanted.