I was tracking down an unknown timeout problem in our cluster and found something interesting.
We currently have 30 pods for a service. When I run kubectl get event
Those 30 pods have been failing for readiness. All of them have "LASTSEEN" less than 10 mins and they are keep failing.
However, I can still access the service without a problem.
I thought Kube removes those pods that failed readiness from being accessed.
Why can I still access the service? I've doubled checked that every single one of them are still failing every 10 minutes or so.