We've seen some odd behaviour on our kubernetes cluster. We have two test applications that speak gRPC. One sends a subscription message and the other sends back a stream of responses. The envisaged behaviour is that this stream stays up until cancelled. However, we were finding situations in which the server thought it was sending updates but the client didn't receive them. Narrowing this down further led to a reproducible test case:
- If the service is configured as gRPC in Kubernetes i.e. grpc- in the port name.
- You set up the streaming
- You then reboot the server
Then the observed behaviour was that the client never saw the connection drop, presumably because its connection is to the istio proxy and not the destination server. However, without the information that the connection has dropped, it can't re-establish the connection.
Has anyone seen this behaviour? What do we need to configure in ISTIO to work around this? (We can fix the problem simply by changing the service to have a port name that doesn't begin with "grpc-" but that works by disabling ISTIO's gRPC functionality.)
Edit:
Kubernetes: 1.14.6 Istio: 1.3.6
There's no explicit DestinationRule set up, although various things have been tried we couldn't find anything that changed this behaviour.