Possible to configure (Squid) proxy so a browser can bypass a VPN?

I think that the best would be to change nothing to your current setup but to take advantage of Linux namespaces instead. You can use tools like nsjail or Firejail for convenience.

The idea would be to set up a dedicated namespace for that other browser that should bypass the VPN, with its own routing rules. That's all you need, change the routing rules so that it uses your regular Ethernet interface rather than the tun interface created by OpenVPN.

The other benefit is isolation of your applications. By limiting their scope and the resources they are able to see, you effectively isolate them from each other.

Using Firejail here is how I would do it roughly - I encourage you to read the doc to fine-tune the configuration to the desired result. Firejail comes with ready to use profiles for common applications so it can immediately start sandboxing your browsers and other applications.

There is a also GUI configuration tool (firetools) but my suggestion would be to try Firejail with just one application instead of reconfiguring your whole environment.

Here is a PoC using Python (assuming you have the netifaces package installed).

firejail --net=enp4s0 --noprofile python3

>>> import netifaces
>>> netifaces.interfaces()
['lo', 'eth0-11182']

As you can see, Python sees only one interface (in addition to the loopback interface). eth0-11182 is of course a virtual interface that exists only within the current namespace.

Suggested reading: Firefox Sandboxing Guide

I haven't used squid, but to my understanding it's just a regular proxy, so by installing it on your computer, it still follows your OS routing rules.

I would try the following setup:

1. Configure OpenVPN to not change your default gateway
2. Install squid or some other proxy service on the same machine running your OpenVPN server

After having this setup configured, your browsers would not use the VPN by default, and you can configure one of them to go via the proxy server using the server's OpenVPN subnet address.

Note1: You have to use the proxy via the OpenVPN subnet address and not its public IP.

Note2: Do not allow traffic to the proxy server port from outside of the OpenVPN subnet, can be achieved using iptables, or by not opening the port using your cloud provider rules

UPDATE:

When using an external VPN service (e.g: nordVPN) you don't have access to the machine running the OpenVPN server and that makes the above solution invalid, since you can't install anything on the machine running the OpenVPN server.

It's possible to install the proxy service in another device in your local subnet, and some router (e.g: mikrotik) provide this feature. But that will make the solution to work only on a specific location with this setup.

Thanks for all the ideas and comments so far. I have not (yet) got firejail to work with the VPN active (@Anonymous suggestion). But I followed up on the suggestion linked by @ofirule specifically this answer to use control groups.

That answer has a link to a complete shell script to do everything automatically but I wanted to do things manually at least to start with so I could follow along and make sure I knew exactly what I was changing. So, following the Manual HowTo there I will post below exactly what I did for anyone else.

Note that my network interface is named enp0s31f6 and my gateway IP is 192.168.0.1 I think those are the only two specific things that may need changing for others following this:

# Install required tools
sudo apt-get install cgroup-lite cgroup-tools

sudo su

# Define a control group named novpn with a classid of 11:11
cd /sys/fs/cgroup/net_cls
mkdir novpn
cd novpn
echo 0x00110011 > net_cls.classid

# Add mark 11 on packets of classid 0x00110011
iptables -t mangle -A OUTPUT -m cgroup --cgroup 0x00110011 -j MARK --set-mark 11

# Force the packets to exit through my interface (enp0s31f6) with NAT
iptables -t nat -A POSTROUTING -m cgroup --cgroup 0x00110011 -o enp0s31f6 -j MASQUERADE

# Define a new "novpn" routing table
echo 11 novpn >> /etc/iproute2/rt_tables

# Packets with mark 11 will use novpn
ip rule add fwmark 11 table novpn

# Add a route for default gateway for novpn
ip route add default via 192.168.0.1 table novpn

# Unset reverse path filtering for all interfaces
# My PREVIOUS VALUES: lo was 0, others (all, default, enp0s31f6) were 2 in case I need to revert back.
for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > $i; done

exit

# Create control group    
sudo cgcreate -t $USER:$USER -a $USER:$USER -g net_cls:novpn

# Run app on specific novpn interface (close all browser windows first)
killall firefox
cgexec -g net_cls:novpn firefox

I don't (yet) fully understand all the above but it works! :-) Running firefox via the cgexec command I can check my IP in firefox with the VPN active and can see it reports my actual public IP not the VPN IP. I also proved this method works by running a radio streaming service which was having issues working with the VPN - it now works absolutely fine bypassing the VPN.