What hardware for powerfull server setup

What kmarsh said, plus

• 64bit, with more than 4gb of memory
• A Remote Console solution (ILO/RSA - NOT something like VNC - you need hardware level remote access) if you're going to locate the server somewhere you can't physically get to it 24/7
• Multiple NICs (at least 4), that allow NIC Teaming
• The most important bit: Maintenance Contract (4/8/12hour response time etc)

Any server:

• Should have ECC memory
• Should have 2 power supplies
• Should have at least a mirrored pair of disks or 4-disk RAID 5 with hot spare plugged in.
• Should have room for growth of RAM and storage (external OK), for example an slot free for an FC card or free SAS ports and room for more disks.
• Should be on a UPS with more than sufficient runtime to a.) endure a typical power outage for your area (say, 10-15 minutes) plus time to shut down (another 10-15 minutes).
• Should have a backup solution integrated from the start (can be external)

The fastest servers are rarely as fast as the fastest gaming systems, nor do they need to be. Servers are more about capacity and throughput, while gaming systems are more about reducing GUI latency. Indeed, most servers have fairly lame graphics hardware.

Form Factor

I think that you'll have better luck with what you're talking about if you go for a rackmount server. However, those are noisy(they typically live in a datacenter where people aren't, so it's not usually a problem).

UPS

You'll want a UPS & possibly a generator outside for those times when the power goes out for a while, although depending on reliability of power, you might be able to get by with just a UPS. APC has some good ones, and they even have a sizing guide.

Hard Disks

kmarsh mentioned RAID 5(3 disks) although RAID 1(2 disks) would also work. Going from 1 to 5 is a speed boost.

Power Supplies

kmarsh also mentioned redundant power supplies. You should definitely get those.

Network

Dual NICs are also a requirement. Linux can bond two NICs together so that it looks like one address. You'd then connect these to redundant switches/routers/firewalls so that either of them could go down without affecting uptime.

Hot Swapping

You mention that you'd like to change faulty components without rebooting; that's known as hot-swapping. That's commonly found in a rackmount.

For all the peripheral components(power supplies, HDs, DVD/CD drives, any floppy drives, & memory), you should check with the manufacturer that they're hot-swappable. Hot swapping has to be supported by the host OS, the motherboard, & the component you're swapping. You mentioned Linux; that does support hot swapping. We use Dells for all of our hardware; they have pretty good support. Also, the support we have will make sure that everything is hot-swappable.

Remote Access

We also have Dell RACs in our servers; the Remote Access Cards allow you to remotely power on/off/reboot the server, connect to the console, or put in a CD without being there. It's like being at the console without driving/flying out there. About the only thing you can't do(that you would be able to onsite) is to replace faulty hardware.

I recommend a DL100 or DL300 series from HP. Get dual power supplies, hardware RAID controller, lots of RAM, and some nice disks. You'll be able to swap disks and power supplies without downtime, and having dual power supplies allows you to move to different power circuits / UPS feeds without downtime too. It will also come with an iLo interface for out-of-band management magic. Those are the main features of "server-class" hardware, IMO.

Next, buy a nice quarter-height rack with some integrated fans, and put a small UPS in the bottom of it. Throw away all of your DLink gear and buy rack-mountable SOHO routers and switches. Buy a variety of lengths of Ethernet patch cables, and a bunch of velcro cable ties to make it all purty. APC has very nice stuff for this.

Your best bet is to find used hardware on Ebay, actually. In fact, that's what we use where I work - we don't have an enormous client base, so there's no great need for super-powerful, cutting edge stuff. What we do need however, are servers that are super-reliable. Most of our servers are various models of Dell Poweredge servers. 2450's are what we use for our DNS servers and our own website and even that is overkill. But they're also very cheap and fairly easy to find. We have a 6650 for our hosted websites and that does its job well, even with several popular sites that use CMS's like Mambo.

Yes, you'll need a good UPS and cooling and backups. The UPS might also be something you can snag on Ebay, but you'll have to make sure the batteries are new. I have no idea where you could get a bargain on air conditioning though. We use a consumer-grade computer in a server case for our backup machine (hard drives are cheap, tape drives are expensive, and you can't automate them worth a damn anyhow). The backup machine could go up and down like a yo-yo and your customers won't notice. And if the drive fails (or if you really like, the software RAID array) then you have until your next data catastrophe to replace it. 6 hours of downtime on this box is also something your customers won't notice.

Now, the bill for all this... will likely be more than a beer budget. For this reason, I don't really know why you don't go in for formal training, because the tuition would be about as much, and you'd probably get to work with newer hardware. Moreover, there's nothing to it. This kind of hardware is so reliable that you hardly even need to interact with it, and you just end up going by what the manual says. The software on the server will be much more complicated, in fact.

For play at home (and your learning process), buy used as Ernie and some others have said and by all means use it at home. It won't be quiet - most proper servers sound like a jet engine when they are starting up.

For anything where you are hosting for your clients on the Internet however, renting is always better than buying. You aren't going to have a diesel generators, multiple power grid connections, multiple internet connections, N+1 UPS, N+1 cooling, everything.