As they point out in a pretty neat article about the issue at Icrontic, the problem is that the industry standard for measuring battery life is a program called MobileMark 2007. This program basically runs your computer through some productivity apps, which are pretty easily cached into RAM on modern notebooks so you don’t get much hard disk usage. These run until the battery dies, and that’s your battery life benchmark. Wi-Fi is almost always disabled, the laptop is almost always in its most power-saving and low performance profile, screen brightness is usually at 50% or less, etc.

There are plenty of advocates for changing this, to use different ways of measuring battery life in notebooks. Most call for a new set of procedures revolving around real-world application tests and new logos or stickers to describe battery life. Frankly, I think most of these ideas won’t go very far. Testing labs want a reliable, repeatable, “set it and forget it” test. We don’t need to rewrite the book on battery life testing, we just need a new version of MobileMark.

What should MobileMark 2010 look like? First, it should reflect what people actually use their notebooks for. This means mostly browsing the web (the modern, dynamic web, not static HTML), some document creation and editing, a little video, and a little 3D gaming.

First, the standard should be for notebooks to be set the way normal people set them. Power profile set to “balanced” or whatever the middle-of-the-road general setting is. Screen brightness at 80% or 90% (which is how most users have it set). Wi-Fi enabled. In fact, Wi-Fi should be used for some of the test.

The test itself should be a 10-minute loop that runs through several typical laptop scenarios. Why 10 minutes? Any shorter and you spend all your time loading tests, not running them. Make the test too long and some notebooks won’t go through enough loops of the test to average out the tests over time. A 15 or 20-minute test might work, but anything longer than that is probably too long.

The first 4 minutes should be a web test. Today’s dynamic web sites are the #1 use of notebooks. BAPCo should host several mock websites (accessible only to registered MobileMark users) that the notebook will load. They mimic the javacript and Flash heavy environments of modern popular sites like Gmail, Facebook, Twitter, Digg, etc. The test can use whatever browser is set as the computer’s default, but it should actually interact with the sites, virtually “clicking” on buttons and opening menus and such.

The next 3 minutes is standard document creation. Load up World, Excel, Powerpoint, and some light image-editing app and run through a scripted scenario of searching, typing, copy/pasting, etc. This is straightforward stuff, and is what MobileMark already does.

The next two minutes is all video. Full screen, hi-def video in a popular format like H.264. If the laptop has a graphics chip that can assist with GPU decoding, that’s fine, it can be used. If not, the video might be jerky and drop frames and such – which is fine. We’re measuring battery life, not quality or performance.

The same goes with the last minute of the 10-minute test. This should be an intense 3D game, or rather, a test made to mimic the load of one. Full-screen 3D graphics that actually puts a load on typical notebook graphics processors, together with some physics and AI routines running a canned animation. It doesn’t need to actually be interactive, it just needs to load up the CPU and graphics chip like a game would. It should be reasonably forward-looking, too; it doesn’t matter if Intel’s integrated graphics gets 4 frames per second while a mid-range discrete GPU from ATI or Nvidia gets 40. Again, we’re just measuring battery life here.

As with the current MobileMark 2007, this loop of tests would just repeat until the laptop dies. I’m guessing your typical “four hour” laptop of today would die in about two hours of this new MobileMark, which is in line with the reality where most users’ laptops die in half the promised time.

This hypothetical new MobileMark 2010 would be a vast improvement over what we currently have, without making the testing labs for notebook manufacturers and editorial press have to make major changes. They would just load on a different version of MobileMark, and the checklist of pre-test conditions (Wi-Fi state, energy saving mode state, screen brightness state) would slightly change. It’s not more work for anyone, it doesn’t mean changing stickers or labels or building a new logo program or any of that sort of stuff.

Of course, it won’t happen unless everyone agrees to jump at once. The need for a more realistic battery test has existed for a long time, but manufacturers don’t want to adopt a standard that makes their notebooks look like battery life is shorter. Certainly, they don’t want to do so unless everybody else does. Yet another reason why a new version of MobileMark with newly mandated test conditions is the best way to go – it’s the path of least resistance to broad industry adoption.

Every site on the ‘net is going to test it’s speed relative to this or that setup. Some will use whatever machine they have lying around, some will construct machines to hit specific spec targets. This ZDNet article does a pretty good job. This browser comparison on BetaNews is somewhat questionable (mixing feature support and standards compliance tests with performance tests to single scores). DailyTech, for some reason, only used 32-bit (when 64-bit is so popular these days).

One thing none of the benchmarks show, but you hear again and again from those that use Windows 7, is just how much more “responsive” and “snappy” it feels. The time between when you click and something happens is noticeably diminished. You really notice it when you go back and use a machine with XP or Vista on it. Unfortunately, this is the kind of thing that benchmark applications simply don’t show, and is very hard to measure. Don’t take my word for it…ask any tech journalist who had to set up and run Windows 7 vs. Vista vs. XP benchmarks for a recent article. Every one I’ve met (and this is the kind of company I keep) will say that numbers aside, Win7 feels much faster. As intangible as that may seem, it’s worth keeping in the back of your mind when you see miles of bar charts comparing Office performance and PCMark Vantage scores and whatnot.