Explaining the Proposal: The Server Hardware

In The Proposal, we specified the following server system:

Server hardware and software: $720

In Humble Beginnings, the fictional Bob Bossman asked us to build out his infrastructure ‘on a shoestring’. Well, part of the reason he did that comes from the real world: ITcookbook operates as a hobby, from our own pockets. We’re interested in sponsorships, but here at the beginning of things, we haven’t proved we can do anything yet, and so we be surprised if anyone gave or loaned us thousands of dollars worth of hardware and software for our curious little venture. So the real world reason for the budget hardware is: we’re not made of money. But that’s alright, the real world imposes budgetary hardships on system administrators millions of times per day. How we deal with those hardships and still create usable and reliable systems is just one of the many ways we bring craftsmanship and professionalism to the table.

So, it’s a cheap server, built specifically to run ESXi (our ESXi choice is explained here). And you might actually be able to buy an ESXi certified one, built and warranted, from a top-tier vendor, for not much more money. In fact, at the time of this writing, we know you can get a Dell T410 tower server, with one hard drive and a gig of RAM, starting at $750 – and it is on the ESXi Hardware Compatibility List (HCL) right here, certified to run ESX 4.0, ESXi 4.0 Installable, ESX 3.5 U4, and ESXi 3.5 Installable U4.

But I needed to construct one out of parts I had, and I wanted this article series to be based on real-world experience. So going forward, as you read our upcoming articles and watch our videos, you’ll know exactly what hardware it was running on. I therefor constructed the FictionalBiz/Bob Bossman scenario to fit hardware I had in my home lab. Did I mention how I’m not made of money?

The thing about ESX/ESXi is, these are very stripped-down operating systems. One way VMware kept their hypervisor a lean, mean, stability machine was by removing the user’s ability to add drivers. So it will work only with that hardware which is on the HCL, whose drivers are included in the product. For home builders like me, this can be a little frustrating! Looking through the VMware Communities, you discover quite a few people learning about ESXi’s short hardware compatibility list the hard way. I was one of them.

However, through much research, trial, and error, what I discovered was this:  to run ESXI with 64bit guests, you want a 64bit virtualization-aware CPU, and your NIC and storage controller must be on the HCL. Everything else (motherboard, CDROM, case, disks, video controller, etc)  will pretty much ‘just work’.

So I first determined what CPU I wanted. I set myself a budget of $50, and picked the best virtualization-aware CPU (and heatsink/fan assembly!) I could find at that price. That meant it had to support either AMD’s AMD-v or Intel’s VT instruction set. As of this writing, all AMD CPU’s have AMD-v, and only some Intel CPUs have VT. And at the time I built this system (nearly a year ago now), the AMD chips in my price range were the best performers. So I ended up with the AMD ‘Brisbane’ CPU listed above.

I would have liked to buy a motherboard with supported NIC and storage controller onboard. But at the time I built the system, the only such motherboards available were very high priced server motherboards – or only available in a server from Dell, HP, or IBM. So my fallback strategy was to obtain compatible NIC and storage controllers as add-in cards, and add them to a consumer-priced motherboard.

So my next job was to find a storage controller that was on the HCL and within my budget, and would support at least four drives. In virtualization, disk bottlenecks are the most common, and so you want as many disks (‘spindles’) as possible – the rule of thumb is at least one spindle per guest VM. My plan was to use commodity SATA drives, so I ended up with the LSI MegaRAID controller. There were quite a few of them on eBay at the time, working pulls from HP servers. Even so, my most careful shopping left me paying about $155 each for these cards.

Next I needed a network interface card (NIC) which was on the HCL – and I wanted one that had dual gigabit copper interfaces. The Intel Pro/1000 MT was a great bargain - $30 each, brand new, on eBay (they retailed around $180 each while I was shopping).

Next, I wanted any motherboard with the following characteristics (in order of importance):

  • Under $80
  • AM2/AM2+ socket for the Brisbane CPU (and upgrade to Phenom later, perhaps)
  • support as much RAM as possible, because the more VMware guests I have, the more RAM I’ll need.
  • PCI-Express-x4 slot for the LSI storage controller (more on that in a moment)
  • PCI/PCI-X slot for Intel dualport nic (more on that in a moment)
  • video onboard, or PCI slot for video card I have lying around

After performing multiple searches at Newegg, I found the ECS motherboard which supports 32 gigabytes of RAM. It only has four memory slots, so I’d need 8-gig RAM sticks to max it out. Those were prohibitively expensive when I set up the system, but the rule of RAM is: prices always fall. So I knew I could buy 4x 2-gig sticks today, and hopefully replace them with larger sticks if/when the prices come down. I paid $10/gig to add 8 gigs of RAM to the system. Not bad!

Lastly I needed a few incidentals – case, video card, CD/DVD reader. I already had a rackmount case, very similar to the Norco one listed. A home computer lab can get quite messy with desktop/tower cases, and years ago I was fortunate enough to bring home a 24u 4-post enclosure which my employer was throwing away. So I can rackmount most of my lab gear, and that’s a blessing. In the workplace, rackmount servers should be considered essential, in my opinion. It costs a little more, but it’s more than worth it because it saves space and helps enforce tidiness. In a future article I will share a few thoughts on why this is important.

I won’t bore you with the details of the generic video card and CD/DVD reader I used. They’re simple commodity items. ESXi will work with basically any video card you throw at it, as long as the card supports VGA mode. Same goes for CD/DVD – as long as the BIOS can find it, ESXi can use it.