One of the nodes in my Kubernetes v1.16.3 cluster has ended up in a strange state. The node is running Debian 10.1 and Docker 19.03.1. When I try to kubectl exec sh
into a container on that node I see these error messages in the node's journalctl -u docker
(the 2nd is also reported by kubectl
itself):
stream copy error: reading from a closed fifo
Error running exec ... in container: failed to create runc console socket: mkdir /tmp/pty...: no space left on device ..."
At this point I don't see which device may have become full. A simple df
does not raise any alarms. How can I further diagnose and remedy the situation?
BTW, the problem so far has persisted across kubectl drain
, systemctl restart docker
, restarting kube-proxy
and the CNI's pod, as well as several other measures.
As OP already mentioned in comments, a combination of
docker system prune
anddocker rmi $(docker images -q)
remedied the situation.