L2TP Client Public IP inaccessible after connecting to VPN

I have an odd issue that i have not come across before.

I have a Fortinet firewall and i have a remote access L2TP VPN configured through the VPN Wizard, works great. The L2TP Client has a public IP address, its a VPS. While it is off the VPN, i am able to ping its public IP just fine from my firewall and internal LAN. As soon as i connect my VPS to the L2TP VPN, both my firewall and internal network are unable to ping the public IP of the VPS BUT, any other external computer like my cell phone or my work computer can access the VPS just fine. Only my firewall and LAN are affected.

I made sure that the VPN routes do not have priority over the public interface default route.

When i run traceroute <public ip> on my lan computer, it just times out giving me no info.

As soon as i disable the L2TP VPN on the VPS, connection is restored...

I don't even know where to begin troubleshooting with this one... Any ideas?