How do I tell when/if/why a container in a kubernetes cluster restarts?

You can view the last restart logs of a container using:

kubectl logs podname -c containername --previous

As described by Sreekanth, kubectl get pods should show you number of restarts, but you can also run

kubectl describe pod podname

And it will show you events sent by the kubelet to the apiserver about the lifecycled events of the pod.

You can also write a final message to /dev/termination-log, and this will show up as described in the docs.

Beside the previous answers another command that helped me to find an error is:

kubectl get event [--namespace=my-namespace]

It lists events from Pods, Jobs, Nodes too

I follow these steps to define the reason for failure:

# print out a pod logs (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs)
kubectl logs {name_of_pod} -n {namespace} --since=2h --timestamps


# print the logs for the _previous_ instance of the container in a pod if it exists
kubectl logs -p {name_of_pod} -n {namespace} --since=2h --timestamps


# check events (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe)
kubectl describe pod {pod_name} -n {namespace}

# look at the 'Events' at the end of the output
# ..
# Events:
#   Type     Reason   Age                 From               Message
#   ----     ------   ----                ----               -------
#   Warning  BackOff  40m                 kubelet, gke-xx    Back-off restarting failed container
# ..


# observe all events (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get)
kubectl get events -n {namespace} --sort-by=.metadata.creationTimestamp


# check logs, etc. in pod container (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec)
kubectl exec -it {pod_name} -n {namespace} -- sh

kubectl get pods will actually list any restarts of the container also the describe command can be of help cause it lists any events associated with the pod.

Liveness probes and readiness probes can be configured for better handling check here

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Additionally hooks can be configured to be consumed in the container at specific points in the life cycle of the container check here

https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/

When restarted

kubectl describe pod your-pod-name

Look for a section like this:

State:          Running
  Started:      Wed, 23 Jun 2021 23:52:05 +1000
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Wed, 23 Jun 2021 23:46:48 +1000
  Finished:     Wed, 23 Jun 2021 23:46:52 +1000
Ready:          True 

The interpretation of the above is as follows:

• The pod was terminated Wed, 23 Jun 2021 23:46:52 +1000 after having started at Wed, 23 Jun 2021 23:46:48 +1000, and is now running and ready, having been last started at Wed, 23 Jun 2021 23:52:05 +1000

A pull request has now been merged into the kubernetes 1.22 milestone, to add LAST RESTART column to kubectl get pods, and will be available once that's released - see here. https://github.com/kubernetes/kubernetes/pull/100142

To see your current version - kubernetes version

(1.21 is the latest release as at 28th June 2021)

If restarted

kubectl get po [your-pod-name] The pod was restarted at some stage if there is any number in the RESTARTS column

Why restarted

kubectl describe pod [your-pod-name] will show a Last State which gives you a high level indication. To see what happened on the pod before it restarted, use kubectl logs your-pod-name --previous. You can pipe this to a file for inspection e.g.

kubectl logs your-pod-name --previous > pod_previous_log.txt

(See also above under 'When restarted')