Following last Wednesday’s announcement and VMworld Europe presentation regarding enterprise production-class hypervisor evaluation criteria, I have received an enormous amount of feedback from both users and vendors. My inbox was flooded with highly positive feedback from more than 150 users (e.g., architects, directors, integrators, and administrators). Vendor feedback, on the other hand, was mixed.

Between the time of the presentation and today, I’ve had the opportunity to speak directly with three of the vendors covered in the presentation, and had an email dialogue with the fourth vendor. Most vendors thought the presentation was fair and balanced, but not all were thrilled with the media coverage. For example, here are two articles in which the announcement was covered:

• VMware outshines Hyper-V, et al. in hypervisor comparison
• Hypervisors Compared: VMware Stands Alone

In the “VMware outshines Hyper-V” article, I was quoted as saying Microsoft would not be happy with the presentation. For background, when I was interviewed by Bridget (the reporter who wrote the article), I had given examples of how each of the four vendors covered in the presentation would not like the results. I mention this because I don’t want it to appear that I was singling out Microsoft. All vendors were treated equally in the presentation, and I didn’t hold punches with any of them. Several folks on the Microsoft virtualization team were present during my presentation, and they were not upset at all. The Microsoft team understood our position and appreciated the fact that I mentioned in the presentation that Hyper-V on Windows Server 2008 R2 will be close to meeting our enterprise production hypervisor feature requirements. Once the product ships, we will conduct a full evaluation against our criteria and go from there.

Citrix XenServer 5.0 and Virtual Iron 4.5 also had respectable showings in the presentation. Sure no vendor wants to hear their shortcomings aired in a public forum, but both vendors, like Microsoft, agreed with our points. That being said, we will stick by our position – only hypervisors that meet our entire list of required features are enterprise production ready. That leaves only VMware VI 3.5 as a product we will recommend for enterprise production roles. As other hypervisors satisfy our feature requirements (one in particular is very close), we will recommend them as well.

Both Microsoft and VMware have come forward and made statements in favor of the evaluation criteria.

“Burton Group is the first technology analyst firm to arm customers with an in-depth list of criteria for evaluating virtualization software. This analysis shows the depth and breadth of Burton Group research expertise, and their ability to deliver substantial value to customers.” – Bogomil Balkansky, VP of Product Marketing, VMware

“As was the case with my colleagues, we thought Burton’s report was a fair representation of the primary commercial hypervisors available to customers. And as we discussed last week, we aim to prove that our collaboration with Citrix, Novell and Red Hat, as well as new features in WS08 R2 and SCVMM r2, will improve Microsoft’s position in Burton’s report.” -Patrick O’Rourke, Group Product Manager – Windows Infrastructure, Microsoft (this quote was taken from this post’s comments)

Of course, user validation is most important, but it’s also important for vendors to accept our criteria and to strive to meet or exceed it. In my opinion, receiving positive backing from the top two vendors in the x86 virtualization space further validates our research and resultant evaluation criteria.

I’m not going to revisit the points in the presentation (you can view a 15 minute summary presentation here), but I cannot emphasize enough the importance of the long term commitment associated with server virtualization platform decisions. Exit costs associated with migrating hypervisors can be significant and must be weighed. Migrating from one virtual hard disk format to another may require the time and nearly the same manpower that it took for the physical to virtual conversions that drove an organization’s initial server virtualization deployment. Also, you’re not just getting a hypervisor, you’re getting an ecosystem, and that is why our evaluation criteria focus heavily on factors such as storage, network, security, and management integration. Hypervisors must provide the performance and stability expected by the organization, but also integrate well within the rest of the IT infrastructure.

Many of our clients considering going with multiple hypervisors see a natural divide between server virtualization and desktop virtualization. You maintain separate VM OS images and rely on different management tools in the desktop space anyway, so using a different hypervisor for the desktop environment is a possibility. In most cases, servers and desktops are managed by different divisions in the IT department. The separation is natural in these cases, and the familiarity with associated management tools is just as important. In addition, the refresh cycles, patch cycles, and other updates are typically different between server and desktop. That being said, vendors looking to penetrate established VMware clients need to do a better job talking about how they are better platforms for virtual desktops. One way to do that is by letting their features do the talking; adding memory overcommit support would be a good first step.

At Burton Group, we will be spending considerable time on competitive differentiation in 2009. The VMworld presentation and announcement was really just the start. If you have thoughts on our evaluation criteria, I would love to hear them.

Note: Originally posted to Burton Group’s Data Center Strategies blog.

March 5, 2009 update: Added VMware and Microsoft quotes.