My company runs filters on contents available for browsing and I have to check if everyone is respecting the browsing policies. Recently I have spotted some users activate OpenVPN on their workstations and i have discovered that they connect on port 443 of an external server and then use a proxy to circumvent the company filters. Is there any method to monitor VPN connections on allowed ports or to avoid the connections?
Do we take it that this OpenVPN connection is not set up by you/your company, and that the end users are connecting to their own OpenVPN server? If OpenVPN is using the standard ports you can certainly block them: drop or reject traffic to TCP and UDP ports 1194, perhaps from specific machines (the users in question) and to specific machines (the OpenVPN server). As it's not (sorry for missing that first time) your question then becomes "How can I distinguish between legitimate HTTP-over-SSL-on-TCP-443 and disallowed OpenVPN-over-SSL-on-TCP-443?". A tool like ssldump might reveal some difference between HTTPS and OpenVPN but turning that into a firewall rule to reject traffic to a particular server might be difficult.
OpenVPN uses SSL for encryption, and breaking that is non-trivial, so you cannot see what's in the VPN traffic from the firewall or other 'gateway' host, however a decent packet sniffer (wireshark) will at least tell you there is SSL traffic between workstation A and public internet address B. You could install traffic sniffers on user's desktops (again you've not said what platform, but I'll assume Windows) directly, and sniff the 'tap' device (there are privacy considerations, especially if people are allowed to use VPNs for light personal use).
Arguably this is as much a policy violation/personnel problem as much as a technical issue, so getting bosses/human resources people involved wont be a bad idea, i.e. get them to announce/reiterate the policy and indeed that technical measures will be taken to reinforce this policy. Do check first that the users are genuinely using OpenVPN to bypass content filtering and what they're accessing - i.e. if they're doing it to access work-related stuff then you should consider whether your filtering policy is of genuine benefit to the company, or it's simply an impediment to getting work done such that people are having to expend time and effort to work around it.
This seems like a personnel issue, and not a technical one. Your organization should be performing regular security audits of your workstations. It should be made clear that if any user is caught using unauthorized software including VPN, proxies, etc, they will be sanctioned.
Really, there's not an 'easy' way to tell the difference between HTTPS over port 443 and OpenVPN. It's doable, but requires more effort than it's worth. If you see an extraordinary amount of traffic, or something else tips you off, maybe it's best to have an 'education session' with your users.