I have a VPS hosting account, and the hosting is offered by Xen, and controllable via VNC. I could connect to my machine with my iOS app, but couldn't with Finder on Mac OS X. Maybe there is an unknown problem on Finder's VNC client. However I need Xen compatible VNC client for Mac OS X. Please recommend some.
It seems that OSX's built in VNC client does something that Xen doesn't. We got a VNC viewer called 'JollysFastVNC', and it works very well with Xen. In fact, I think it's the only OS one that I've gotten to work correctly.
Are you able to ping your VPS from MacOS? And can you connect to the VNC on the VPS with a Windows or Linux client? This sounds more like a basic network connectivity issue than a VPS issue. Or your VNC server is not configured correctly.
If other devices can access VNC then you should just try other software. Have you tried RealVNC? Chicken of the VNC? Jollys Fast VNC?
I really like Chicken of the VNC. Cant imagine it wouldn't work.
http://sourceforge.net/projects/cotvnc/