Use secondary IP as outgoing IP on SOCKS5 traffic that uses this IP

I have a server that runs OpenVPN on it's primary IP.

There is also a second IP I connect to using SSH as a SOCKS5 setup like this:

ssh -C2qTnN -D 1337 me@<secondary-ip>

I go to whatismyip.com from each connection and I want to see the primary IP when I connect through the VPN and the secondary when I connect through the SOCKS5 proxy (assuming my browser is using the proxy obviously) but now always the primary IP shows.

What trickery do I need to do to fix this?

Edit: this is on a vps and the different IP addresses are on interfaces venet0:0 and venet0:1