For the past couple weeks, I have had the pleasure of working closely with VMware Corporate Architect Lance Berc while testing his ESXi hypervisor PXE boot solution. There’s been plenty of good solutions for PXE installing the ESX hypervisor. In fact, Mike Laverick has written a great how-to on the subject. However, when it comes to PXE booting and configuring ESXi, so far we have been left in anticipation.

There are plenty of advantages to PXE booting a hypervisor, including:

• Rapid hypervisor deployment to nearly any server
• Rapid hypervisor updates – a hypervisor’s VMs can be live migrated (e.g. VMotion) to a new server and their original host rebooted to receive an update
• Servers can be deployed without any internal hard disks, making them more energy efficient (the PXE-booted hypervisor would run in physical RAM, removing the need for a physical hard disk)

Virtual Iron realized the benefits of this concept a long time ago and has offered PXE boot support of their Xen hypervisor implementation for years. Considering ESXi’s small size (32MB), offering a PXE boot option for ESXi makes a lot of sense. Some ESX gurus have been working on solving the ESX PXE boot riddle for quite some time (see Vinternals’ PXE Boot ESX 3i Installable Edition for more details). However, it was VMware’s Lance Berc that was able to develop a fully working proof of concept that includes deploying a new ESXi hypervisor and configuring all necessary settings (e.g. storage, vSwitches and physical NICs, cluster association). I’ve had the opportunity to test and validate Lance’s solution in my lab, and with his permission, I’m able to disclose a rough outline of the details. It’s my hope that by disclosing the VMware proof-of-concept that enough interest will be generated to propel PXE booting ESXi as a future product feature.

Now let’s get on to the details. For starters, you’re going to need a PXE server, and each ESXi server will need network interfaces and a system BIOS that supports PXE/network boot. If you have a system so old that it doesn’t support these features, then it’s probably not a good candidate to run ESXi anyway.

In addition to the PXE server, you will also need:

• VMware ESXi boot and image files extracted from the ESXi installable CD (binmod.tgz, environ.tgz, cim.tgz, oem.tgz)
• A Virtual Center server
• A PXE boot midwife
• Microsoft PowerShell (installed on the VC server)
• VMware PowerShell toolkit
• VMware SDK Perl toolkit
• lance.tgz
• Midwife scripts (download Lance’s samples here)

Lance’s PXEextract document does a nice job detailing the steps needed to extract the ESXi boot and image files from the ESXi installation CD or .iso image. The PXE boot midwife runs the scripts necessary to remotely configure the PXE booted ESXi server; in essence, it assists in the birth of the server. Here is Lance Berc’s description of the midwife’s role:

Introducing the new machine to the environment has several steps:

1. Boot ESXi via DHCP/PXE, passing the address of the midwife
2. After initialization the new system sends a birth message to the midwife
3. The midwife cleans residual state from Virtual Center
4. The midwife uses Virtual Center to configure the new system

We have built a prototype midwife out of scripts utilizing the VMware VI3 SDK.  There are two sets of scripts, one in Perl and the other in PowerShell. This is very raw code and has not been made robust. 

Microsoft PowerShell, the VMware PowerShell toolkit, and the VMware SDK Perl toolkit had to be installed on the VC server (also the midwife) in order to run the midwife scripts. lance.tgz contains a few files that I’ll let remain nameless. As you can suspect, that package is needed to get the stateless configuration to fully run.

Once you have everything in place, deploying a new ESXi host is a piece of cake. In my lab, I was able to automatically configure an ESXi box and join it to an existing VC cluster. Thus far, I’ve booted stateless ESXi hosts with both NFS and iSCSI storage without a hitch, and plan to test a PXE boot ESXi box with Fibre Channel storage in the coming weeks.

PXE booting ESXi opens numerous hypervisor management and deployment possibilities that previously didn’t exist. If this is a feature you would like to see in a future VMware release, please let me know via a comment to this blog. Let’s consider this a virtual petition for requesting a necessary new feature in VMware Virtual Infrastructure deployment and management.

I’m guessing that some of you may want to be able to test this solution in some capacity as soon as it is available. If that sounds like you, keep your eye on Lance Berc’s posts in the VMware forums.