Developers on Reviewing Video Cards

Tim Sweeney
Author of the UnrealEngine, Epic Games

I think your site has the right approach. What you're trying to do -- evaluate hardware performance in the context of future upcoming software -- just happens to be really hard. Developers aren't going to provide you with benchmarks at least until they're reasonable close to shipping, and that's a good thing, since benchmarking hardware with very much unfinished and unoptimized software is going to produce unreliable results.

"Use synthetic benchmarks, analyze the hardware architecture abstractly, talk to developers, write your own benchmark code to test particular features. You're doing the right stuff." Tim Sweeney Epic Games

But I think you're doing the right thing for consumers. Most enthusiasts go 18-24 months between video card upgrades, so the most important thing about a card isn't its performance in current stuff, but in games shipping over the next 0-18 months.

So, what can you do? Use synthetic benchmarks, analyze the hardware architecture abstractly, talk to developers, write your own benchmark code to test particular features. You're doing the right stuff.

Unreal Tournament 2003 A DX7/DX8 game

Long-term (over the next 2-5 years) this mess will go away. The moves from DX6 to DX7-8 and DX9 have been marked by radical changes in hardware architecture. To take full advantage, developers have had to radically rearchitect their engines, as we did going from Unreal1/UnrealTournament to UT2003/UT2004, and are doing again for the move to our third-generation engine.

The move from DirectX9 to future architectures will be much smoother. Though performance will continue to improve significantly each year, the basic feature set will only change gradually -- once you have support for computationally complete shaders, the evolution of GPU's will look much more like that of CPU's, with changes being more subtle and performance-oriented rather than feature oriented. The tradeoffs will then be things like number of pipelines, cache size, number of floating point units, which is much less dramatic than the questions of hardware T&L, 8-bit integer vs 24/32-bit floating point, etc.

I don't know if there's a specific question except "what benchmarks we should use" in your mail, but I think a right mix of synthetic and game apps are the best way to go - and have always been. The rest is called professional journalism.

"... I think a right mix of synthetic and game apps are the best way to go - and have always been. The rest is called professional journalism." Markus Maki Remedy/Futuremark

I appreciate the effort and attention to detail that goes into creating your reviews, and feel that your site is unique in how you do things. I have stopped visiting other sites that do a 16-page review with 100 different benchmark graphs and (if we're blessed with luck) a one-row analysis of it all. When I'm reading about something, I want detailed analysis or well-formed opinions, not just some benchmark spew.

This analysis can in my opinion be done either from the technical analysis standpoint, i.e. looking at the hardware specs, functionality, theory etc. in detail, or semi-subjectively, looking at the end user experience by playing games, looking at out of the box experience etc. Both are valid approaches, but perhaps for a different group of people.

Max Payne 2 A DX8.1 game

In both cases, I feel everyone focuses too closely on the tiny details such as AA quality or aniso quality/performance, not on the larger picture. Is this card good now? Will it, in a generic sense be good with future games? And why?

Also, both approaches need integrity, which in some examples I've seen depends a lot on where the advertising dollars come from, unfortunately...

But as to better ways of reviewing products, I don't think there are any. If there are "revolutions" in 3d graphics like HW T&L and DX9 shaders, identifying those is the first step, so proper weight can be put to test and analyse those features.

3DMark03 A DX9 Benchmark

When I was at Futuremark, we were in the exact same conundrum when developing 3DMark (and it's exactly like this every year); 3 manufacturers said they'd ship a HW T&L product in 2001 and only one did so. Still, it was clear that it was a "must have" feature, although the benchmark got a lot of flak for being pro-NVIDIA. Only history will prove you right or wrong, and people have very short memories.

But in my opinion, you have the skills to be right - and the connections to ask for help when you're uncertain.

Nice to hear from you. It may be that it's even a tougher problem than that. Even if I would use Splinter Cell - X engine to benchmark video cards right now, it wouldn't be exactly fair. Let me explain:

"So, I guess I'm changing my mind a little about how useful synthetic benchmarks are. They could be much more useful at predicting future game performance than real game benchmarks." Dany Lepage Ubisoft

During the pre-production phase of a video game, you work as hard as possible to include as many features as possible in your engine so artists/level designers can use these features to make the game. Features that are coming late are either not going to be use as much or don't require as much involvement from the assets creator. This explains why there are so few real per pixel games out there (with normal maps), it changes the whole production pipeline and that need to be done from the beginning of production or previous assets are going to be wasted.

Practically, what this means is that we rush features in and we barely think about optimization for a while. Would that be worthwhile to benchmark video cards (or CPUs) with this kind of engine? In my opinion, the answer is no.

Splinter Cell A DX8.1 game

Of course, even if we end up having a Shader Model 3.0 rendering pipeline, the game won't have been designed from scratch with the SM 3.0 in mind and it will be an extension (artist assets are going to stay the same as before) of the same architecture (nothing revolutionary that couldn't been done before, just better and faster).

So, I guess I'm changing my mind a little about how useful synthetic benchmarks are. They could be much more useful at predicting future game performance than real game benchmarks. However, I think developers would really need to get involve in the process. I don't see much incentive for us to do that right now but maybe we should do it...

On the other hand, it will take some time (probably at the end of 2006) before we start seeing games based on engine that were developed with SM 3.0 in mind.