A few weeks ago, we provided a look at the major options for Windows virtualization on Mac OS X: Parallels Desktop 8 and VMware Fusion 5. A head-to-head comparison showed that Parallels 8 beat Fusion 5 overall, although performance was very close in most benchmarks.
Many readers asked us to perform additional testing to determine the improvements between Parallels 7 and 8, and how the free VirtualBox software compared to the paid options. Readers were also curious about native Windows performance in Boot Camp.
We’ve spent a few days testing and we now have our results. Read on for the ultimate performance comparison between Parallels 7, Parallels 8, Fusion 4, Fusion 5, Virtual Box, and Boot Camp.
Hardware & Software
Our tests were performed on a 2011 27-inch iMac at 3.4 GHz with 16 GB of RAM, running OS X 10.8.1. We used Fusion 4.1.3, Fusion 5.0.1, Parallels 7.0.15107, Parallels 8.0.18108.797180 (we have to talk to Parallels about a simpler versioning scheme), and VirtualBox 4.2 to virtualize Windows 7 Professional 64-bit.
All virtual machines were stored on and accessed from an external Pegasus R4 Thunderbolt RAID array with four 3 TB 7200 rpm hard drives in a RAID 5 configuration. We could not install Windows via Boot Camp on this drive, so native Windows was installed on the iMac’s internal 240 GB OWC SATA III SSD.
All virtual machines were configured to use four of the iMac’s eight logical processors and 4 GB of RAM. The configuration options don’t completely translate between applications, but each was set up to maximize the performance of the virtual machine over that of the host OS.
For the Boot Camp tests, all benchmarks when possible were set to use only four cores to provide a more accurate comparison.
Testing Methodolgy
Except where otherwise noted, each test was run three times in each configuration and the results, as long as they were within five percent of each other, were averaged. If any result was abnormal, we retested until we could determine the source of the abnormality and then discarded that result. Not all tests were possible in all configurations. These exceptions are noted for each test.
Futuremark’s PCMark benchmark attempts to evaluate overall system performance, and includes tests for computational tasks, image and video playback and manipulation, web browsing, gaming, and storage speed.
It should first be noted that some virtual benchmarks, such as the Lightweight and Productivity tests, report artificially high scores. This is due to the inability to have an "apples to apples" CPU comparison between virtual and native hardware, and to certain virtualization optimizations that can inflate synthetic benchmark scores.
Also of note is the Boot Camp result for the Computation test. PCMark 7 does not allow a user to restrict the number of CPU cores, so the Boot Camp tests were able to use 8 logical cores while the virtualized tests were restricted to 4. Normalized Boot Camp performance would still be better than virtualized performance, but not by as large of a margin.
Other than those tests, Boot Camp is the obvious and expected winner, although virtualized performance is not far behind in some categories. As we demonstrated in our previous testing, Parallels 8 offers the best virtualization performance for this test, with Parallels 7 in second place and Fusion 4 and 5 nearly tied for third. The free VirtualBox has a long way to go to match the performance of the commercial software applications and comes in a distant fifth place among virtualized options.
If we focus just on the current versions of the virtualization software compared to Boot Camp, we can see that Parallels 8 holds a roughly 10 percent advantage over Fusion 5, while native Boot Camp tacks on about 21 percent higher performance over Parallels 8.
Futuremark’s 3DMark06 is a DirectX 9 gaming benchmark that attempts to stress a system’s GPU and CPU the same way that a highly detailed game would. It’s a bit old at this point but it provides one of the best ways to test DirectX 9 performance, for which both Parallels and Fusion offer full support. VirtualBox has basic 3D support but would not run the 3DMark06 benchmark.
The results show the same trend found in the PCMark 7 tests: Boot Camp takes first, Parallels 7 and 8 lead the virtualization results, and Fusion 4 and 5 finish last.
Again focusing just on the current versions of virtualization applications, we see that Parallels 8 scores about 8.5 percent higher than Fusion 5, and about 11.5 percent lower than Boot Camp.
Futuremark’s 3DMark Vantage is a DirectX 10 benchmark. While the iMac’s GPU supports DirectX 10 natively in Boot Camp, only Parallels 8 supports the multimedia API while virtualized. Parallels classifies its DirectX 10 support as “experimental,” so improvements in performance should be expected over time.
3DMark Vantage has various presets for its CPU and GPU tests. We used the default settings for the Performance (P) and Entry (E) tests.
The results show that DirectX 10 support has a long way to go. While it is functional, gamers should not expect to be virtualizing their favorite DX10 games any time soon.
Geekbench is a multi-platform tool for measuring a system’s computational and memory performance. It does not test graphical or storage capabilities, but is useful in that it scales from systems as small as an iPhone to those as large as dozen-processor workstations.
Geekbench can be run in either 32- or 64-bit mode. As we were using a 64-bit version of Windows with 4 GB of RAM, we ran the 64-bit mode.
Again, Parallels has the advantage over Fusion in most tests, although both applications come close to native performance in the memory and stream tests. The free VirtualBox holds its own in this test as well. It doesn’t win any category, but it does keep within ten percent of Parallels on most tests. Boot Camp holds substantial leads in Integer and Floating Point calculations.
An examination of the overall Geekbench score for the current versions of virtualization software tells the same story as previous tests and ranks performance in the following order: Boot Camp > Parallels 8 > Fusion 5 > VirtualBox 4.2.
Cinebench is a multi-platform benchmarking utility that is based on Maxon’s Cinema 4D rendering software. It tests OpenGL graphics performance and multi- and single-CPU rendering capabilities.
Like the GPU tests above, VirtualBox could not run the OpenGL portion of the Cinebench test. It did run the rendering tests, and so it is included in the chart with a zero for its OpenGL results.
Parallels 8 scored the highest of the virtualization applications on the OpenGl test with 41.4 frames per second, but all virtualization software fell well behind native performance at 74.65 frames per second.
It was a much tighter race in terms of rendering performance. We limited Boot Camp tests to four processors but performance was still slightly better than virtualized options. Parallels 7 held a surprising, albeit slight, lead among virtualization software and VirtualBox scored very well compared to its commercial peers, taking second place in the multi-core test.
Just Cause 2 is an open-world action game released by Eidos Interactive in 2010. It supports DirectX 10 and so, like our 3DMark Vantage test above, was only tested on Boot Camp and Parallels 8 via its experimental DirectX 10 mode. We tested four resolutions on the “Desert Sunrise” built-in benchmark.
Parallels 8 offers playable frame rates, but falls far short of native performance. There also appears to be some issues with Parallels’ DirectX 10 driver, as resolutions between 1440x900 and 800x600 scored roughly the same. As we advised above, gamers should not yet plan to enjoy DirectX 10 gaming from this generation of virtualization software.
Crysis, released by Crytek in 2007, was once the ultimate measure of a gaming PC's performance, bringing even the most expensive and capable machines to their knees. In a sign of how far we have progressed in the past five years, Crysis is playable in a virtualized environment on a Mac. Competitive gamers who demand at least 60 frames per second will be disappointed, but more casual gamers who want to see if their Mac “can play Crysis,” will be able to enjoy the game.
Using the Crysis Benchmark Tool, we ran the medium quality test in DirectX 9 mode at three resolutions.
Boot Camp has a huge performance lead but a surprising second place result is Fusion 4. As noted below the chart, however, there were noticeable graphical glitches during the test. It was still playable, but considering the glitches and the performance of the other applications, Fusion 4’s results do not carry much weight.
Between Parallels 8 and Fusion 5, results were nearly identical with the exception of the 1280x720 resolution, where we think Fusion hit a virtualized processing limit, as its score is the same as 1680x1050.
Mafia II, released in 2010 by 2K Games, is a third-person open-world adventure game. It runs in DirectX9 but would not run properly on Fusion 4 or 5, either crashing before completing the benchmark or freezing and stuttering throughout the run. We have therefore omitted those applications from the chart.
Again, Boot Camp provides substantially better performance, but the game does run at playable frame rates in Parallels. Parallels 8 increased frame rates by about 12 to 15 percent over Parallels 7, although we think there is a bottleneck in the way Parallels virtualizes the GPU, as frame rates were roughly the same across the bottom three resolutions.
Boot Times
Using a clean installation of Windows 7, we timed five cold boots for each application with a stop watch and then averaged the results, rounding to the nearest second.
The timer started when we clicked the power or start buttons to launch the virtual machine from within each application, and it ended when all items in the Windows system tray were loaded.
Parallels 8 scored the fastest boot time at 19 seconds, followed by Fusion 5 at 22 seconds. Parallels 8’s time represents a 42 percent increase over Parallels 7.
VirtualBox took the longest to boot, at 38 seconds, nearly twice the time as Parallels 8 and Fusion 5. As long as users plan to keep a single virtual machine open for a while, however, 20 additional seconds of boot time should not severely impact a user’s experience.
Conclusions
In light of our recent tests, there should be no surprises in today’s results. Users who need the absolute best performance should natively boot Windows using Boot Camp. Those who want a compromise between performance and convenience should consider Parallels 8 or Fusion 5, although Parallels 8 holds a performance advantage in most tests.
Users looking for a free solution can try the free VirtualBox, although its significantly lower performance underscores the adage that “you get what you pay for.” Depending on the desired task, however, VirtualBox may be all some users need.
It should be noted that these tests look only at raw performance. Both Parallels and Fusion have a bevy of additional features, many of which are exclusive to one application or another. Raw performance is not the only factor that should be considered when choosing a virtualization application, and users should check out each application’s feature list to see the absence of certain features are deal-breakers for their usage needs.
In the end, Parallels 8 and Fusion 5 improved on the performance of their direct predecessors, and Parallels 8 wins in overall performance.
Users interested in either application can download free trials from the Parallels (14 days) and Fusion (30 days) websites. VirtualBox is available for free from Oracle. Boot Camp is included in OS X. All applications can be used to virtualize many different operating systems including free distributions of Linux, although those looking to virtualize Windows will need to obtain a physical installation disk or disk image of the OS.