vSphere Home Lab: Part 1

After getting the VCAP5-DCD exam out of the way I started to work out what hardware I’d buy for creating a new home lab for my DCA study. Up till now I have used my main gaming rig as a home lab, running an 8-core AMD FX cpu and 32GB of RAM. While this has served me well it isn’t ideal and doesn’t have the flexibility I’d like.

I started trawling through numerous blogs about other home labs and liked the idea of using the Supermicro uATX motherboards that support the E3 Xeons and IPMI. However, after a lot of looking mostly on Amazon (here in NZ the only place I could find boards from was going to cost me almost $400 NZD per board…) I gave up. It was going to be too risky ordering pc gear from overseas and not have the confidence I’d get the right memory modules, etc. Don’t get me wrong, I’d love to have some, in particular the MBD-X9SCM-iiF as it has the two onboard 82574L LAN ports as well as the dedicated IPMI port. But for what I needed I could not justify almost doubling my budget, particularly as the E3 Xeons, such as the E3-1230 would set me back almost $400 NZD a piece too.
Instead I opted for more AMD based gear πŸ™‚
Here is the spec I came up with:

3 x AMD FX 6100 Six-core CPU 3.3ghz – $180 NZD each

3 x Gigabyte GA-78LMTUSB3 – Nice uATX form factor, supports FX cpus, can take up to 32GB DDR3 with support for ECC un-buffered RAM – $115 NZD each

3 x Coolermaster 343 uATX cases (these are pretty cheap and are reasonably small) – $97 NZD each

6 x OCZ Vertex2 120GB SSDs – I got these because they were on special for $114 NZD each πŸ™‚

6 x 8GB DDR3 1333mhz non-ECC – These were about $65 NZD each. Couldn’t afford to go with ECC and didn’t feel I really needed it…when money permits I’ll be upgrading each host to 32gb RAM

3 x HP NC364T 4 port GbE NIC’s – I’m using some spare ones from work

2 x HP ProCurve 2910al-48G switches – Another loaner from work πŸ˜› We had these surplus and aren’t planning on deploying them anywhere

3 x HP P4000 VSA licenses – Yet another thing I was lucky to get from work, we had three licenses we purchased a while back and ended up putting physical P4300 SAN’s in so I figured these would be perfect in a home lab!

Here’s a few pics of the gear so far. Excuse the poor quality photos…my HTC Sensation’s camera is not working that well running a beta JB ROM πŸ™‚

HP Procurve 2910al-48G

HP switches – sweet!!!!

My three vSphere hosts

Cool, VCAP-DCA here I come!

The guts

Cheap and cheerful, no frills at it’s best! Notice I haven’t installed the additional NIC card or the SSDs…where’s my adapters!!!!!

All up I’ve spent close to $2500 NZD which isn’t too bad, but certainly not a cheap exercise…oh well, it’s going to be a great tool for learning so it’s worth it for that!

Bear in mind that most of these parts won’t be on the VMware HCL but this isn’t a production environment, and as such they don’t need to be.

So, I’ve got all the gear mostly built other than waiting on some 2.5″ to 3.5″ SSD drive adapters (the cases don’t have 2.5″ bays 😦 ) and I screwed up with one of the cases. I accidentally purchased the wrong model (I initially purchased only one case as a test) and didn’t realise that the power supply included didn’t have a 4+4 12v molex plug for the cpu power…argh! I’ve got an adapter cable coming that will fix the problem though. I also have three 4gb USB sticks on order too for the hypervisor to boot from. This will mean I can allocate as much of the SSD storage as possible to the VSA’s.

At this stage I think I’ll configure the VSA cluster volumes using NRAID5 (for those of you who haven’t used the HP Lefthand gear it supports various network RAID levels when using multiple nodes) as this will give me close to 400GB of SSD storage. I’ll enable thin provisioning on both the datastores and in the VSAs so I should get a reasonable number of VM’s on it.

If you are wondering “but what about TRIM support?” I have thought about this. It seems that vSphere does not support the TRIM command but to be honest I don’t really care. I figure it will probably take me a while to kill them and they do have a three year warranty :-). At one stage I was going to build a FreeNAS server or similar with all the SSDs (which does support TRIM) but I thought I’d get more out of running the VSAs. Since I use P4300 SANs at work this would give me more opportunity to play around with the software and different configurations.

As for the network configuration, I haven’t quite decided my layout yet. I am probably going to trunk two nics for Management, vMotion, FT and VM traffic, possibly leaving two nics for iSCSI. I probably won’t get the same benefit out of using two nics per host for iSCSI as I would with a physical SAN as the VSA only supports one virtual network adapter (i think…it’s been a long time since I looked at it) but I will still be able to simulate uplink failure, etc.

Anyway, I better get back to trying to configure these switches…went to plug my rollover cable into them and realised my pc doesn’t have a serial port…doh!
Stay tuned for part 2, building the hosts and setting up the VSA πŸ˜‰

Advertisements

About Ben Loveday
My name is Ben Loveday and I am working as a Systems Architect in New Zealand. I have a keen interest in VMware products and are VCAP5-DCD, VCAP5-DCA and Microsoft MCITP certified. I am studying towards VCDX5 certification…I hope! My areas of focus are the virtualisation of manufacturing automation systems with the aim to improve traditional automation/SCADA system design and improve their availability and reliability. I am married with three kids and my hobbies include playing the guitar (less often than I’d like) and listening to music, mostly 80β€²s Metal and Rock/Blues :-) Oh..and I'm a PC gamer!

21 Responses to vSphere Home Lab: Part 1

  1. Gabi says:

    You have too much spare money πŸ™‚

    Excellent setup, really good.

    Looking forward to the rest of your info πŸ™‚

    G.

  2. Ian says:

    where did you see that this board supports ecc ram?

    • Ben Loveday says:

      Hi Ian,

      Most AMD cpus support ECC RAM as of the old Athlon X2’s I believe. Generally the motherboard does not come into play with AMD’s as the memory controller is integrated into the CPU. This is something I believe is now also the same with most new Intel procs except desktop that Intel CPU’s don’t generally support ECC. Bear in mind that the AMD Athlon/Phenom/FX CPU’s don’t support registered or buffered ECC RAM. Because of this the RAM DIMMs are generally smaller in capacity and more expensive. The last time I checked a registered ECC 8GB DIMM was about $80NZD whereas an unregistered ECC 8GB DIMM might cost around $120NZD. The way I understand buffered/registered RAM is that the memory chips contain buffer registers allowing the memory controller to support larger amounts of RAM on the same number of channels.
      You are right in that Gigabyte don’t specify that the motherboard supports ECC RAM but it should work, unfortunately I don’t have any to test this theory though πŸ˜› If I get my hands on any I’ll give it a whirl.

      Cheers!
      Ben

      • Ian says:

        Not sure what is involved in purchasing from the USA for you, but this might interest you

        http://www.superbiiz.com/detail.php?name=W1333EB8GM

        I read that some gigabyte boards do not “officially” support ecc ram, but that the ecc function will be enabled if ecc ram is installed. For this board in question, as expected, the answer I got from gigabyte support was that it does not support ecc ram. Will the ecc function be enabled or not with ecc ram, who knows. This is where I was hoping you had actually tried this.

        In any case, even though the cpu can support ecc, additional pins on the ddr3 dimm also have to be connected (eg, cb 0-7, err_out, etc…). Kind of hard to tell though, with a multilayer pcb. I suppose one could probe the cpu socket and dimm socket for connectivity.

        I decided not to chance it, and am going to get an asus m5a97 r2 board.

      • Ben Loveday says:

        Hi Ian,

        I think I may have come across some unbuffered ECC at work so will try and give it a whirl this week. They are only 4GB DIMMs but should be enough to prove whether the boards even work with ECC RAM!
        Thanks for the link to the SuperTalent RAM, I’ve ended up with non-ECC and should do the trick just fine.
        That ASUS board does look nice, I must say I prefer ASUS boards. My main pc is an ASUS Crosshair V Formula board which is pretty awesome, just way too overkill for my homelab hosts πŸ˜›
        Let me know how you get on with your gear! I’m slowly getting everything configured so that it can run something!

        Cheers,
        Ben

      • Ben Loveday says:

        Hi Ian,

        I just thought I’d let you know, I got my hands on some unregistered ECC DIMMs (4x4gb) and tried them. They definitely work!!! The award bios has an additional advanced memory configuration that seems to be enabled when ECC is detected. It seems to have quite a few ECC specific options which is cool. As expected registered DIMMs definitely DON’T work πŸ™‚

        Anyway, I hope your lab gear is coming along well.

        Cheers,
        Ben

      • James says:

        Hi Ben,

        I’ve been looking for information related to the Gigabyte GA-78LMT-USB3 motherboard / ECC RAM and discover your blog after days of searching on the net. Can you confirm that the motherboard indeed support ECC ram?

      • Ben Loveday says:

        Hi James,
        Thanks for your post. Yes, I can confirm that the motherboard does in fact support unbuffered ECC RAM. I am using 4GB 1066 ECC unbuffered sticks in one at the moment. When the board detects the ECC RAM the BIOS reveals ECC related settings that are not normally visible. I haven’t tried 8GB ECC unbuffered DIMMs but I imagine they’d work if you had some.
        Hope this helps!
        Cheers,
        Ben

      • mbcbmmgccgm says:

        Thanks for the tip about ECC support, Internet stranger. Oddly, this page seems to be the only place on the internet that confirms whether or not the board supports it.

      • Ben Loveday says:

        Hello fellow internet stranger πŸ˜› Tell me about it!! I looked for ages and found nothing, so after finally testing it myself I figured I’d post the information πŸ™‚
        Thanks for reading!

      • Ben Loveday says:

        No worries Barry! I’ve always liked the FX series for that reason, too bad they don’t support registered DIMMs eh! Cheers, Ben

  3. Danny says:

    So Ben you can 100% confirm that the Gigabyte GA-78LMT-USB3 Motherboard along with the AMD CPU FX-8150 (or FX-6100/FX-8120) works with VMware ESXi 5.1??

    Ive been searching the net all day and spec’d this system up. I searched the motherboard name and vmware and up came your site!

    • Ben Loveday says:

      Hi Danny,

      Yes mate. While it isn’t on the official HCL the board works just fine. You won’t get any support for directed I/O and the board doesn’t have IPMI but it is a very cost effective option!

      Cheers,
      Ben

  4. mike says:

    hi Ben,
    I have a bit of a cheeky request, perhaps you can drop me a mail and maybe help a fellow vmware enthusiast out πŸ™‚
    cheers

  5. Pingback: VMware nested vs. physical lab

  6. Jay says:

    I have this same motherboard but when I try to load up esxi 5.5 it says it doesnt see a nic, I’m trying to load it with the integrated nic. Any ideas?

    • Ben Loveday says:

      Hi Jay, thanks for your comment. You’ll probably find that the default ESXi image does not contains Realtek drivers so you’ll either have to try adding the drivers into a custom ISO or install a separate nic card (as I had) which is natively supported. Hope this helps and good luck with the lab! Cheers, Ben

      • Jay says:

        Thanks Ben, yea last night after I did as much research as I could come to find out that I would have to inject the driver into the .iso. Probably doesn’t take to much to do but I’m just gonna throw 5.1 ESXi on it and see. I’m pretty sure it has the driver, thanks for the blog. I’ll stay tuned!!

      • Ben Loveday says:

        Cheers dude πŸ™‚

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: