This might seem like a ridiculous question, but mainly I'm wondering how can I test the difference in latency between the ISP we use in our office and the hosting provider we use (Rackspace in this case). Like what would be the difference in speed if we hosted our servers in our office as opposed to at Rackspace. Can I just pick an arbitrary place in the US and ping it from each and receive a realistic representation of the speed difference?
How fast should our ISP be to compete with one of those big dudes? is 50mbps fast enough?
In general, you don't host with a provider solely for speed. A company like Rackspace with dedicated servers (I know they don't do much colocation so I'm assuming you're using a dedicated offering) is giving you:
You don't host with a provider like Rackspace for the purpose of raw connection speed. That's like asking how many RPMs you'd need to hit in your Toyota in order to match a Ferrari. It's certainly a metric, but it's meaningless without context.
Now, you may be able to provide some or all of the above yourself (and there are many unmentioned factors), but the real question when looking at self-hosting, colocation, dedicated offsite servers, "cloud" hosting, etc, is reliability, performance, and efficiency per unit cost, and what is an acceptable level of each. There are no easy answers.
Ultimately, though, to answer your question, on a pure speed basis you're at the very least going to need a dedicated line à la T1, or some other symmetric connection. Your typical home-style connection, and indeed many "business" hookups, just don't have the outgoing capacity, especially not in a reliable, non-bursty manner.