Beyond3D's 3DMark03 Opinion
Game Tests
With Game Test 1 I am personally ambivalent to the type of scene Futuremark and its Beta participants have picked. If they feel this type of scene is representative of current gaming titles then it may deserve just as much of a position in the benchmark as many others.
However, given the performance of current boards on GT1 you can already begin to question how well this will scale with future graphics boards, bearing in mind that we may see up to three or four generations of graphics boards in the lifetime of this benchmark. Our testing showed that there is quite a steep performance rise from the low to the high resolutions and the performances across many of the boards start quite close which would tend to indicate the benchmark is already fairly system limited at low resolutions. This system limitation will likely only be amplified over time as 3D performance has tended to scale quicker than system performance, which is what we have seen with prior benchmarks.
One issue that I do have with Game Test 1 that I mentioned to Futuremark during the beta testing is with the use of Vertex Shaders in this test. If the test is going to be a primarily DX7 benchmark then why not use hardware Transformation rather than Vertex Shaders? When I quizzed Futuremark about this their response was along the lines that this was supposed to be a DX8 & DX9 benchmark, but there were still plenty of DX7 titles going to be released in the near future, so they appear to have taken a middle ground approach. I'm not sure that this was the correct route to take. If they wanted a DX7 test, to be representative of the games coming up, in my opinion it should be a pure DX7 test, or if they want 3DMark03 to be a DX8/9 benchmark then why bother with the test at all?
As far as Game Tests 2 & 3 go, I have no issues what they are rendering or how they are rendering it; both of the scenes look good and are representative of many of the effects and rendering techniques that we are likely to see in upcoming titles. NVIDIA appears to suggest that this isn't valid because it's not a gaming engine, but does everyone buy a middleware engine? Not really, many developers are still making their own these days and these tests should not be precluded because they have not adopted an 'off-the-shelf' engine or one known to be used in upcoming titles (this may still occur). In our introduction to 3DMark03 I commented that this may actually be closer to a gaming environment, and part of the reason for that was the fact that Futuremark have decided to use an 'off-the-shelf' middleware physics engine in the form of Havok and the use of the physics engine moves them away from being a pure synthetic benchmarks and close to a gaming environment.
Where Game Test 2 & 3 fall over a little is the fact that are just too similar. The trends between boards shown in the 3DMark03 are virtually exactly the same between each of the scenes, limiting their individual usefulness a little.
As far as GT4 is concerned, again, I don't have much issues with the tests itself -- it's a nice looking scene with some of the most realistic water rendering effects seen to date. I'm not sure if this scene is particularly applicable to many gaming titles, but this is meant to be seen as more of a DX9 demo in my opinion. However, I've asked Futuremark and they inform me GT4 was coded using hand written assembly rather than DX9 HLSL, which I don't know will be the norm for future titles as developers start to target shader capabilities more; it may be in the interim, but possible not for the majority in the longer term. Perhaps more complex shaders may have been utilised had they been written with HLSL.
In 3DMark2001 the first few game tests also featured both high and low detail settings, during my feedback as a beta tester I lamented the fact that this option no longer existed on the full game test suite.
So, to sum up on the game tests, I'm ambivalent to the use of GT1, I think GT2 & 3 good scenes with numerous rendering techniques that will be used in upcoming titles (and are being taught to developers by both major IHV's) and GT4 is a nice DX9 introductory scene. However, is this forward thinking enough? Many of the rendering techniques in GT2 & 3 are going to be used in titles such as Doom III and Halo, so in essence these are representative of near term titles, and the DX9 test possibly could have made more far reaching use of DX9 by adopting the use of HLSL.
If I have no particular care about GT1 that probably means that, in fact, I think it should not really be there as I'm not really likely to test using it. It's been suggested to me by Lars from Riva Station/Toms Hardware that perhaps GT2 or GT3 should have featured only a PS1.1 rendering path, while the other featured a and PS1.1 and PS1.4 rendering paths -- this seems like a sensible option and it would make room for the potential to make this test more complex (more lights?) thus further differentiating it from the other test. GT4, in my opinion, is fine, though if GT1 were removed there would be room for a more far reaching DX9 benchmark, perhaps coded by the use of DX9's HLSL.
