It is increasingly looking like we will need to host our own servers, because we need modems physically attached to the server machines.
I (think) we will need a T1 line to our office for starters. Dont know what else is involved other than the obvious redundancy and failover requirements.
My questions are:
Do we really have to do it our selves or can we find a service that allows the modems to be remote as well?
If we have to host the servers ourselves, what are the steps (technical and operational) required?
Google shows me 52 200 000 hits for the terms: self host website, the first 10 seem to contain a fair few discussions of the pros, cons and requirements for doing this.
You might also want to look into colocation as this may give you physical access to your server (or a server leased from the hosting company) in a server room operated by a colocation company who deal with all of the connection to the internet side of things.
Many colocation providers will usually let you have additional communication lines (ranging from POTS to ISDN to T1/E1 to dedicated fiber links to other datacenters) at your rack, especially providers who also play in the telecommunication world. Bills will go up very fast, though, for custom links to anything/anywhere that is not their standard network.
Having your systems in a colo will let you bypass lots of problems, so I really recommend it (UPS and power redundancy, connection redundancy and its related BGP or similar IP re-routing stuff, low latency, air conditioning, staff onsite 24/7, etc).
Note, it's really curious why you would want "modems physically attached to the server machines" - and I know you don't mean POTS...
The connection to the internet doesn't necessarily have to be a T1, but it really should provide a dedicated IP address. Also, check the TOS for the provider to ensure you won't be breaking any of their rules running a server on the end point. We use Time Warner Business Class Road Runner just for our web server but it's an Intranet, not really meant for public consumption.
The next step is to get yourself a server, Linux, FreeBSD, Windows, your choice. Just make sure it is properly secured, firewalled and updated. If you don't have administration experience or are unwilling to hire some, stop right here...go back to managed hosting.
Set up your website, your modem, test to make sure it works. Point your DNS provider to your new IP address and you should be good. As for providers willing to hook up a modem, it's not the modem that's the problem, it's the telephone line drop. You might be able to speak with someone at the datacenter and see if they have a way of providing copper and tone.
For your first question, many people have servers elsewhere. They buy server capacity from a cloud provider, pay for hosting on somebody else's system, have their servers physically at some place that offers colocation, or simply have a data center that's separate from the rest of the company. I don't really understand your requirements. Yes, there needs to be a physical connection between the servers and the Internet. Due to the wonders of IP and the domain name system, they can be in lots of different places. If you have special needs here, you haven't articulated them.
For your second question, the first operational step would be to hire somebody who knows what he or she is doing, at least for initial setup and training. Frankly, you don't sound like anybody in your organization is knowledgeable enough to get things done reliably and correctly. I could be wrong, but that's the impression I get from your question.
I will give you a few tips on running a website, or for that matter any business operations:
Without knowing your load requirements, you need: