Synthetic Benchmarks and Games

Tim Sweeney and Markus Maki both agree that not only are synthetic benchmarks useful, they are relevant, should be used in reviews and that they are not just complementary but can in fact be crucial to an understanding of a video card under consideration. Tim makes a good point and observation about the buying/upgrading habits of the general gaming population. Why do you upgrade your video card to the latest ones? Is it solely to play the games you can buy now, at faster framerates with cleaner graphics that anti-aliasing and anisotropic filtering offer? Or do you also hope that future games -- games that will take advantage of the latest technologies featured in the spanking new video card under consideration -- will play at decent framerates on this video card too? If it's just the former, then you will usually experience faster performance than the video card you're seeking to replace, with probably anti-aliasing/anisotropic filtering applied to boot. What about the latter consideration? The smart buyer will almost undoubtedly take that into consideration. It is not only being smart, it's being prudent with money. Dany Lepage basically admits that synthetic benchmarks are important in this money-saving consideration when buying a new video card.

However, as Tim said, gauging the potential of a video card in relation to future software/games is just a hard thing to do for hardware reviewers. As much as Tim may like to, the responsible thing for him to do would be to deny Beyond3D the opportunity to have a look at (and present our observations to the public) the engine he's currently hard at work on now. As he explained to this reviewer when enquired about such a possibility :

It's too early to start thinking about this [...]. Our goal in facilitating benchmarks is to incentivise hardware vendors to optimize for our performance patterns and feature usage, and though lots of stuff is up and running, we're a ways away from having nailed down our final feature set.

UnrealEngine3 is the name for the next major step in Unreal technology, aimed at DirectX9 hardware as the absolute minimum spec, and scaling (way) upwards from there.

So what can a hardware reviewer do if he wants to provide information beyond how a particular video card runs currently available games? Like Tim said, use synthetic benchmarks and make sure you understand what's happening when using synthetic benchmarks. They are useful if you're considering a new video card's potential in games to come (what is called "future-proof-ability" of a new video card). They are almost always not useful if you're talking about games you can buy just after the release of the video card.

Be Careful about Optimizations

DOOM3 id Software

For hardware reviewers, using synthetic benchmarks isn't actually as simple as it sounds. Just like optimizations for games, IHVs do tend to provide updated drivers that are optimized for specific synthetic benchmarks. Drivers have built-in mechanisms for recognizing certain games and software, including synthetic benchmarks, and the optimizations kick-in when such recognized games/benchmarks are run. During the course of a research and the development of a synthetic software that can be benchmarked, this reviewer asked John Carmack of id Software about synthetic benchmarks and their usefulness to developers such as him and John had the following advice for this reviewer :

One point I would make is that it is valuable to have the source to the synthetic benchmark available. Driver vendors will be doing complete call traces anyway, you might as well make the source public so developers can see exactly which paths get what performance, which isn't always clear from a one line title on a performance graph.

I do worry about the incredible number of variables there are to test for over something as simple as texture access -- texture size, texture filtering, texture compression / depth, etc, especially when permuted through large numbers of texture units. Still, synthetic benchmarks do have value, and an open one would be a benefit over commercially supported ones.

A software where its source (i.e. information that shows you how the software is coded) is provided allows you the possibility to alter the original author's codes for whatever reasons you have in mind. One of such reasons could be that you may want to see performance bottlenecks. The other would be to defeat any such "call traces" (as John puts it) or software recognition by drivers that results in optimizations that you may not particularly want. Many of the synthetic software/benchmarks used by Beyond3D are open-sourced and have a definite advantage over such applications as Futuremark's 3DMark series. To defeat unwanted optimizations in order to have different hardware run the same exact codes to provide much more valid comparisons, the software's original codes may need to be altered, either by third parties (open-source software, such as the full version of ShaderMark 2.0, allows this) or by the software's original author (like Futuremark has done several times to its 3DMark03 semi-commercial application via patches). Some software may also feature built-in features that circumvent unwanted optimizations (the aforementioned ShaderMark is again an example).

If you already own one of the latest video cards, you'll probably want drivers that provide various optimizations that improves performance in the games you currently play (with perhaps acceptable image quality sacrifices). If you're looking to upgrade to one of the latest video cards with one eye on its performance in future games then it may be best to observe the behaviour of video cards in synthetic benchmarks, with the one without any known optmizations in its drivers winning in performance usually means it already has a leg up on its competition.