Aliasing

In ray tracing, the rays you fire are a simple mathematical model, not a physically based model, and the mathematical model has the property of zero cross-sectional area; the ray has no size or volume. This is what allows the intersection tests to be solved relatively simply. However this proves a real problem that has a very visible result, aliasing.

Sampling theory covers the reasons for aliasing, but in this situation, the result is fairly simple. Without any concept of area, the renderer can only generate a yes/no result in the intersection test. It's impossible to have a ray that partially intersects an object. However, sampling anything with zero or infinitesimal area will always alias, since it will either contribute wholly to the signal or not. The best you can do without fundamentally changing ray tracing itself is to sample at a higher resolution and hope to move the aliasing out of the frequencies we can see.

There are two solutions used in the off-line world: cone/beam tracing or distributed ray tracing. The first is simply too expensive, as the cost of an intersection test with cones or beams is extremely high. Distributed ray tracing replaces every single ray with a bundle of rays and uses a Monte-Carlo simulation to get an anti-aliased result back. It’s quite simple to implement, but because it uses the "bundle of rays" model, it exponentially increases the number of rays being traced.

Static or moving scenes?

A serious problem that is often overlooked when demonstrating a ray traced game like Quake 3 RT is that while ray tracing scales well for static scenes, it does not for moving scenes. The most important operation in ray tracing is to determine the closest triangle from the origin of any particular ray. This is generally done by building a partitioning tree which allows you to quickly disregard most of the triangles in the scene without actually doing the expensive intersection test. The problem is that computing these trees is expensive, and any change to the location of triangles inside the tree means recomputing the entire partition tree.

This poses a real problem for things like swaying grass and trees and could be potentially more even more catastrophic for performance when animating skinned characters. As games today can easily have 500+ individually skinned characters per frame, the two solutions would be recomputing the partition tree prior to every new frame or using brute force tracing without a partition tree to accelerate the process.