Last Wednesday QLogic announced what appeared to be a very impressive benchmark - QLogic Achieves Near-Native Fibre Channel I/O Performance On Windows Server 2008 Hyper-V. By near native performance, QLogic highlighted throughput of nearly 200,000 IOPs. Naturally such a high throughput in a virtualized environment caught my attention. The announcement was timed to go along with the Hyper-V RTM announcement and immediately validate storage I/O performance of Hyper-V connected to SAN storage using QLogic 8Gb fibre channel host bus adapters (HBAs). I’ve always liked benchmarks if they can set relative expectations for how a particular configuration will perform in a typical environment. When the environment is far from typical, I consider the benchmark either an academic exercise (let’s see how far we can push this thing, regardless of how unrealistic the configuration may be) or a crafty attempt at product marketing. If I was to place this particular benchmark into one of Nik Simpson’s benchmarking categories, I’d have to say it falls into the benchmarketing category.

The QLogic press release included the following quote from Microsoft’s Mike Schultz:

QLogic’s benchmark result surpasses the existing benchmark results in the market, and demonstrates that Windows Server 2008 Hyper-V customers can achieve higher server utilization rates and consolidate servers with great technical performance.

The statement “surpasses the existing benchmark results in the market” implies that the Hyper-V/QLogic benchmark has outperformed a comparable VMware benchmark. The press release was careful to state the hypervisor and fibre channel HBA (QLogic 2500 Series 8Gb adapter), but failed to mention the back end storage configuration. I consider this to be an important omission. After some digging around, I was able to find the benchmark results here. If I was watching an Olympic event, this would be the moment where after thinking I witnessed an incredible athletic event, I learned that the athlete tested positive for steroids. Microsoft and QLogic didn’t take a fibre channel disk array and inject it with Stanzanol or rub it with “the clear,” but they did use solid state storage. The storage array used was a Texas Memory RamSan 325 FC storage array. The benchmark that resulted in nearly 200,000 IOPS, as you’ll see from the diagram, ran within 90% of native performance (180,000 IOPS). However, this benchmark used a completely unrealistic block size of 512 bytes (a block size of 8K or 16K would have been more realistic). The benchmark that resulted in close to native throughput (3% performance delta) yielded performance of 120,426 IOPS with an 8KB block size. No other virtualization vendors have published benchmarks using solid state storage, so the QLogic/Hyper-V benchmark, to me, really hasn’t proven anything. Furthermore, the published benchmark fails to reveal latency numbers, which has been the most useful value of storage performance in virtualized environments. Applications can be very sensitive to I/O latency, and it’s import to disclose latency numbers in any storage benchmark.

For further clarity, I ran these results by a colleague well-versed in performance testing and this was his response:

In a storage stack, the number of concurrent I/Os is typically a limit at certain choke points, i.e., the virtual device, the queue between the guest and the parent OS, and the drivers in the parent. The recent Microsoft benchmark used an I/O depth of just 64, but with an SSD the latency is very small, so at 0.3ms per I/O with an SSD, it’s possible to generate 210,000 IOPS in theory at 0.3ms with 64 outstanding I/Os.

However, to properly demonstrate 180,000 real IOPS would require 1,200 concurrent I/Os, rather than the 64 used.

With real disks, the same 64 concurrent I/Os at 7ms each would limit throughput to 64 * 1/.007 = 9,142 IOPS!

To me, these exercises in smoke and mirrors trickery (i.e. solid state storage in a hypervisor storage performance “benchmark”) yield more questions than answers. In addition, I’m left questioning future benchmarks produced by vendors that use such tactics. Vendors - if you are going to go as far as issuing a press release based on a “benchmark,” please give us an honest assessment of a real world environment. Anything else simply casts doubt on your future performance numbers and adds to the already prolific cynicism surrounding vendor benchmarks.