Tech Note: Mask-based Foveated Rendering with Unreal Engine 4

Mask-based foveated rendering (MBFR) is a rendering technique that decreases the shading rate of the peripheral region of the eye buffers by dropping some of the pixels, based on a checkboard pattern. It allows you to effectively reduce pixel shading cost from world rendering, provides good visual quality when dropping 50% of the pixels, and is compatible with all D3D11 level hardware - independent to IHV extensions.

In this post, we will walk through a masked scene that demonstrates MBFR and discuss when MBFR is most useful.

Here is a masked scene which demonstrates the pixel dropping. There are four rings in the masked image. The center circle preserves all the pixels. The other three rings drop 25%, 50%, and 75% respectively. The pixels are dropped in 2x2 quads.

Here is the reconstructed image:

Here is the original full resolution scene we use as the ground truth:

Here is a side-by-side comparison (magnified). The top is the masked image, the middle is the reconstructed image, and the bottom is the ground truth image.

Here is another comparison:

MBFR is currently included on Oculus GitHub UE4 Repository (GitHub login required). It works with Unreal Engine 4.19+, while you should review our Unreal Get Started Page for Rift if you haven’t already.

To activate MBFR in your project, follow these steps:

Here are some console commands that can be helpful when testing MBFR and tweaking the quality levels:

The pixels to drop are culled through a mask bit in the stencil buffer. Here are the visibility patterns in each density region:

The dropped pixels are reconstructed in the PostProcessing stage, when the scene textures are fetched.

We use the 4 neighboring pixels to reconstruct each dropped pixel in the medium- and high-density regions, where the pixel dropping rate is less than or equal to 50%.

The naïve formula is to set the dropped pixel A to the linear interpolation of B, C, D and E.

But this doesn’t work well and causes a lot of aliasing in the high-contrast / high-frequency areas. The artifact can be easily observed in the screenshots below and caused heavy flickering in VR.

The visual quality can be greatly improved by applying the directional weights computed from the neighborhood luminance. It enhances the edges and reduces the aliasing.

Finally, we achieve a good overall reconstruction quality on the high/medium density regions. Here is a side-by-side comparison on the high-density region (25% of the pixels are dropped), where the center image shows the reconstructed result and the image on the right is the ground truth:

And here are two comparisons on the medium-density region (50% of the pixels are dropped) where the center image is the reconstructed result and the image on the right is the ground truth:

We have fewer known pixels in the low-density region, at the most peripheral area. We reconstruct each dropped pixel with linear interpolation of the two neighbor pixels.

Since we only have ¼ of the effective pixels, the level of fidelity of the reconstruction is limited, with significant quality loss and distortion.

Here is a side-by-side comparison with the pre-distortion images (left: masked, center: reconstructed, right: original):

Because the low-density region is at the peripheral area, where the effective pixel density is low. It reduces the quality loss, but does not eliminate all the distortion artifacts. Here is approximately the same area on the post-distortion images, which has become much smaller after distortion:

The high-fidelity mask reconstruction is GPU-expensive and not every post-processing stage requires this level of precision. So, we also provide a coarse-quality mask reconstruction routine which reduces the reconstruction cost to a single texture fetch, from 2 or 4 texture fetches in the regular reconstruction. The coarse quality reconstruction is currently used in Bloom, Depth of Field, Down-sampling, Velocity flattening, and Screen-space reflection.

MBFR reduces GPU pixel shading cost by dropping a subset of the pixels in the world rendering passes. But it also introduces extra cost in the post-processing passes for reconstructing the dropped pixels.

By testing on a 4.65ms scene of Robo Recall (on GTX1080), it reduced the overall GPU cost to 4.25ms (-9%). Specifically, it reduced the world rendering (BasePass) cost from 2.51ms to 2.19ms (-13%); and the PostProcessing cost was increased from 0.47ms to 0.65ms (+0.18ms).

Since the cost of PostProcessing is relatively fixed in a game, and the world rendering is dynamic, we can expect more than 10% GPU performance savings from a scene with heavy world objects rendering.

Of course, the overall GPU performance is also heavily dependent on the FOVs of the density rings. An effective way to further optimize the GPU cost from MBFR is to configure them according to the content being rendered on each map. We can apply larger density rings in maps which contain high-frequency objects/textures to preserve the visual quality (like the cityscape maps in Robo Recall we tested), and reduce the size of density rings in other maps to receive the greatest performance benefit.

MBFR is not a free lunch. It reduces the pixel shading cost from world rendering but adds extra post-processing cost from the pixel reconstruction. Most VR projects are not using heavy post-processing. But to projects which use heavy post processing but little world rendering, MBFR may not bring any performance benefit.

The perceptual visual quality of MBFR can vary significantly based on the style of the content. The perceptual quality can be quite good when rendering low-contrast, low-frequency contents, but would be less optimal when there are a lot of high-frequency details in the content. Tweaking the radius of the foveation density rings would be necessary to balance the performance and perceptual visual quality according to the content being rendered.