Microsoft bashes Windows XP, Vista benchmarking

Gregg Keizer   Today’s Top Stories   or  Other Windows Stories  

Moving to Windows Vista: The Promise, The Reality Take Control! Managing Your Application Library The Advantages of Migrating to Windows Vista® and Office 2007 Simultaneously The Importance of Application Management Windows Vista®: A Cyber Security Shield Windows Vista: Necessity and Opportunity SaaS Solutions for Remote Systems Management Compterworld Technology Briefing: Intelligent Users Use Business Intelligence Computerworld Report- Large Enterprises Pay More for IPAM Sign up to receive Windows Resource Alerts

December 5, 2007 (Computerworld) -- Microsoft Corp. and a performance testing researcher are trading barbs over recent benchmarks that claim the unreleased upgrade for Windows XP runs Office faster than the upgrade slated to ship early next year for Vista.

According to tests run by Devil Mountain Software Inc., Windows XP Service Pack 3 (SP3) blows through the company's OfficeBench test suite about twice as fast as Vista SP1 does.

Nick White, a program manager for Microsoft's Vista team, bashed the benchmarking in a post to a Microsoft blog last week, and called Devil Mountain to task for testing before either operating system reached its "Release to Manufacturing" (RTM) stage, the so-called gold code that is actually duplicated on disc. "Publishing benchmarks of the performance of Windows Vista SP1 now wouldn't be a worthwhile exercise for our customers, as the code is still in development and, to the degree that benchmarking tests are involved, remains a moving target," White argued.

He also called into question OfficeBench, dismissing it as a "window-open, window-close routine at accelerated speed."

"It isn't representative of real-world user behavior and hence isn't an accurate gauge of the actual end-user experience," White said. "Tests like these only measure a very small set of Windows capabilities and so aren't representative of the user's overall day-to-day experience of working with Windows and running applications."

White's comments raised the hackles of Craig Barth, Devil Mountain's chief technology officer, who called out the Microsoft program manager in more than one place. "As the author of the OfficeBench test script I take personal offense at your cavalier handling of what is a very real and pressing issue for enterprise IT organizations," Barth said in a comment attached to White's blog. "Consider this fair warning that we will NOT let this issue rest and that you'll be hearing from us again in the near future."

"It was a hit-and-run job," Barth added today in a telephone interview. "We're formally challenging them to prove that the [OfficeBench] benchmark results aren't valid. If they can, I'd really like to see it."

Barth defended the test script as realistic and countered White's contention that benchmarks at the service pack stage are meaningless. "I think it was reasonable to test both release candidates at this point," he said. "My experience [with Windows] since 1991 is that if there's a performance difference of more than a percent or two between release candidate and RTM, someone at Microsoft [messed up].

"If this was Beta 2, if this was any beta, I wouldn't have touched it. But pre-RTM, which is what release candidate is, is fair game."

Barth also smacked White for a video clip posted to the Vista blog that showed OfficeBench in action, but at a speed Barth said was wildly accelerated. White cited the video to back up his claim that the test suite didn't measure real-world usage.

Continued...
1 | 2 | NEXT  

Print this Story

Send Us Feedback

E-mail this Story

Digg this Story

Slashdot this Story

"After too much denial, Apple has finally admitted that Mac users need to install anti-virus software. Now it's time for..." Read more... "Todd Bishop at TechFlash had a great summary last week of some e-mails revealed during the Vista Capable lawsuit that..." Read more... Read more Windows posts or See all Blogs

Windows users indifferent to Microsoft patch alarm, says researcher Tech jobs down sharply but not out Apple yanks antivirus advice from its Web site More top stories...

Microsoft slates 8 bug updates for year's final Patch Tuesday De Beers tries to force spoof news Web site offline over fake ad Microsoft confirms Yahoo's Lu to run online services

Review: The new MacBook Air, now with extra SSD goodness Thin as ever, the latest Air offers up to twice the storage and snappy performance. Cool Stuff: Your 2008 Holiday Gift Guide We've got an array of economical, expensive, and just plain weird tech gifts for your friends and family. Massive botnet returns from the dead, starts spamming The spam-spewing 'Srizbi' botnet that was shut down two weeks ago has been resurrected and is again under criminal control, say security researchers. Elgan: Why you can't trust 'friends' on Facebook Facebook is popular and growing -- especially with criminals. Here's why they love it. Windows 7 Get the latest news, reviews and more about Microsoft's newest desktop operating system More Continuing Coverage Windows 7 Voting Technology Google's Chrome browser Economy in turmoil: Latest news and tips iPhone 3G Bill Gates moves on Windows Vista A to Z Mac A to Z 2008 Salary Survey Find wage data for 50 IT job titles. More Special Reports 2008 Premier 100 IT Leaders 2007 Premier 100 IT Leaders 2008 Best Places to Work in IT 2007 Best Places to Work in IT 2008 Salary Survey 2007 Salary Survey 2006 Salary Survey 2008 IT Forecast 2007 IT Forecast 40 Years of Computerworld Top 12 Green-IT Users The FAQs About Going Green IT Schools to Watch iPhone: One Year Later Global Mobile Your Next Data Center Management: Reinventing IT Stop Data Leaks Web 2.0 Security Surviving Disasters Managing Storage Virtualization The Lean Storage Machine Storage on Trial Can Web 2.0 Save BI? Bypass These BI Potholes All Zones Business Continuity Zone The File Data Management Zone Security Management Zone The SAS Zone Business Intelligence and Analytics Zone The Enterprise Search Zone Software as a Service Zone The Security Zone See your link here

Symantec Backup Exec System Recovery-Restore Systems Anytime, from Anywhere to Virtually Any Device Download this white paper, FREE, compliments of Symantec! Restore Windows systems quickly, easily, and reliably to dissimilar hardware, virtual environments or in remote, unattended locations. Watch this Symantec webcast to learn how you can perform bare metal system recovery in minutes, meet and exceed service level agreements and recovery time objectives, and minimize requirements for remote on-site IT support. Download this executive briefing  Advances in SSL and Certificate Management Advances in SSL and Certificate Management View this webcast now! Go to the webcast  Windows Vista: Necessity and Opportunity Download this white paper today! (Novell) The Vista era of Windows is here. Yet most organizations will retain Windows XP alongside new Vista PCs for some time. Your challenge is to manage a mixed environment thus reducing the disruption of change, streamlining systems management and boosting end-user productivity so that your organization can better meet its business goals. Download this white paper  White Papers Read up on the latest ideas and technologies from companies that sell hardware, software and services. 8 Things You Need to Handle Today's Network Traffic Next-Gen Load Balancing: 3 Keys to Successful Delivery of Advanced Web Apps Building a Reliable and Dynamic Data Center with PAN Manager by Egenera View more whitepapers

• Windows users indifferent to Microsoft patch alarm, says researcher
• Microsoft slates 8 bug updates for year's final Patch Tuesday
• Tech jobs down sharply but not out
• More top stories 

Accelerate your pursuit of perfection For almost 80 years, Kodak has been helping banks, insurance companies, healthcare providers, government agencies and other businesses produce billions of document images. So Kodak is uniquely positioned to know and deliver–what customers want: easy-to-use scanners that output the best possible image quality. Download this white paper now!

Protecting Exchange While it was once just a convenient way for employees to communicate internally, today e-mail systems like Exchange are tightly integrated with other business applications and are one of the primary methods for communicating with current and prospective customers. Protecting Exchange against costly downtime has become a top priority for more IT departments. So how do you ensure that your Exchange environment is always protected? Download this white paper now!

Dan Tynan Today's Internet has been brought to you by porn YouTube and Ning are trying to shed adult-oriented content to appeal to mainstream advertisers. Will it work? ... [more]