MS Outlook keeps not responding when requesting data from Exchange

Looking for some pointers on this one…. MS Outlook keeps not responding when requesting data from Exchange

The setup up…

Three locations:

• one is the head office running Windows 2016 servers (AD environment), one of the servers is running Exchange 2016 (Latest CU13) with one CAS. In addition the site is also running ADConnect as the company is using MS Teams from 365. All mailboxes are stored on-premise. The addition of 365 Hybrid is only recent as the company wanted Teams during these current times we find ourselves in. However this issue I writing about today has been happening before the addition of Hybrid Exchange and 365. Note the hybrid exchange was only setup to allow MS Teams App to see the calendars
• one remote sites is connected to the head office via site-to-site VPNs using DrayTek routers. 2 users, so no servers just PCs running in a local workgroup.
• one remote sites is connected to the head office via site-to-site VPNs also using DrayTek routers. 3 users, running a window 2012 R2 servers in a different domain with trust relationships to HQ.

All connections are at least 100Mbs

On the remote sites when running MS Outlook 2016 or 2019; when clicking on an email or changing folders outlook goes to not responding. After a few seconds it comes back to life and works fine, until you change folders again.

Checking the outlook connection status, I can see the connections to the server going via the VPN (using the local internal address, exchange.ad.example.com). Outlook has got this address correctly via Autodiscover.

I checked the following so far:

• The setup with the MS Connectivity tool and nothing noteworthy there.
• I changed outlook from Cached Mode to Online and that made it worse.
• I checked the bandwidth and the traffic peaks at 3%, so that’s not it.
• I checked the event viewer on the clients and the server and nothing noteworthy there either.
• I uninstalled MSOffice and reinstalled, no change

I did however notice on the outlook connection status window some strange Avg Response times. One of the users has access to several mailboxes (all on the same server). The response time to most of the mailboxes is between 43 and 65, but on a couple of others its as high as 622.

Could this be the cause of outlook to not respond? Waiting 622ms? Although it wasn’t the mailbox the user was looking at, at the time. When you disconnect Outlook and connect again the results are different and the long waiting times are now on a different mailbox.

• I tried removing access to the remote mailboxes and using the main mailbox but still Outlook does a not-responding.

Looking at the Req/Fail column there are 0 failures. Most connections say Cached, with a few saying foreground and background.

• I tried deleting the users local profile (and OST file) to rebuild their cached copy and it still does it.

• I checked the firewall settings, and between the sites internally its just the default settings from a typical windows & exchange installation.

• I read somewhere about changing the ActiveSync timeout on IIS, but that didn’t resolve it either.

When the users go into the HQ Office and log in on a hot desk and run Outlook there are no issues at all.

So its only over the WAN (VPN) links.

Any ideas?