Advantages and Disadvantages of Analytical Anti-Aliasing
Compared to sampling-based anti-aliasing methods, analytical solutions have several important advantages. When they work well (on correctly detected geometry edges) they can provide quality that equals sampling based methods with a very high sample count, at a fraction of the computational cost. Furthermore, they are easily applicable in a variety of scenarios that make implementing sampling-based AA more challenging, such as deferred shading.
However, analytical AA is not a panacea. An inherent problem with purely analytical methods based on a single sample per pixel is that they cannot deal with sub-pixel aliasing. If a pixel structure such as the one in the upper right of Figure 2 is presented to an analytical anti-aliasing algorithm, it has absolutely no way of knowing that the disconnected pixel groups are in fact forming a line. At this point, there are two equally undesirable ways to handle the situation: either blur the pixels, reducing apparent aliasing but also destroying detail, or conservatively only handle clearly identified staircase artefacts which are certain to be caused by aliasing, which will leave sub-pixel aliasing unchanged and aliased.
Another issue with analytical methods are false positives. When a part of an image is identified as an aliased edge, but is not actually such an edge, it will be distorted by blending. This is particularly apparent on text, and again necessitates a trade-off: more conservative edge detection will result in fewer false positives, but will also miss some edges that actually are aliased. On the other hand, extending the threshold for edge detection to also include those edges will result in false positives. As analytical anti-aliasing at its core tries to extrapolate more information from a rasterized image than is actually available, these problems are impossible to fully eliminate.
Finally, the interpretation of edges by these methods can differ wildly based on a single pixel difference. Therefore flickering and temporal image instability can be increased or even introduced by single-sample analytical anti-aliasing methods, as a single switching pixel in the original image can turn into an entire flickering line in the anti-aliased output.
Figure 11 illustrates some of the successes and failures of analytical AA methods on the example of the common FXAA and SMAA1x algorithms, the latter of which is generally considered the best purely analytical real-time-capable single-sample algorithm.
