Oculus Rift supports Mask-Based Foveated Rendering (MBFR) which enables the edges of the eye buffers to be rendered at a lower resolution than the center portion of the eye buffers, based on a superimposed mask.
Mask-Based Foveated Rendering (MBFR) decreases the shading rate of the peripheral region of the eye buffers by dropping pixels based on a checkboard pattern.
There are four rings in the masked image. The center circle preserves all the pixels. The other three rings drop 25%, 50%, and 75% of the pixels, respectively. The pixels are dropped in 2x2 quads.
Here is the reconstructed image:
Here is the full resolution scene that serves as the ground truth in this example:
Here is a side-by-side comparison (magnified). The top is the masked image, the middle is the reconstructed image, and the bottom is the ground truth image.
Here is another comparison:
MBFR is included in the Oculus GitHub UE4 Repository. (See the Version Compatibility Reference.) It works with the Unreal Engine 4.19 and 4.20-preview versions.
To activate MBFR in your project:
Here are some useful console commands for testing MBFR and adjusting the quality levels:
The pixels to drop are culled through a bit in the stencil buffer. Here are the visibility patterns in each density region.
The dropped pixels are reconstructed in the PostProcessing stage, when the scene textures are fetched.
Mid/high density region
We use the 4 neighbor pixels to reconstruct each dropped pixel in the mid/high density regions, where the pixel dropping rate is less than or equal to 50%.
The naïve formula is to set the dropped pixel A to the linear interpolation of B, C, D and E:
But it doesn’t work well and causes too much aliasing in the high contrast / high frequency areas. The artifact can be easily observed in the screenshots below. It causes heavy flickering in VR.
The visual quality can be greatly improved by applying the directional weights computed from the neighborhood luminance. It enhances the edges and reduces the aliasing.
Finally, we achieve a good overall reconstruction quality on the high/medium density regions. Here is a side-by-side comparison on the high-density region (dropped 25% pixels; the middle is the reconstructed result and the right is the ground truth):
And two comparisons on the medium-density region (dropped 50% pixels, the middle is the reconstructed result and the right is the ground truth):
Low density region (75% drop rate)
We have fewer known pixels in the low-density region, which is furthest out on the peripheral area. We reconstruct each dropped pixel with linear interpolation of the two neighbor pixels.
Since we only have ¼ of the effective pixels, the fidelity of the reconstruction is restricted, with significant quality loss and distortion.
Since we only have 1/4 of the effective pixels, the fidelity of reconstruction is restricted, with significant quality loss and distortion. Here is a side-by-side comparison with the pre-distortion images (left: masked, middle: reconstructed, right: original):
Because the low-density region is at the peripheral area, where the effective pixel density is low, it reduces the quality loss, but would not eliminate all the distortion artifacts. Here is approximately the same area on the post-distortion images:
Coarse quality reconstruction
The high-fidelity mask reconstruction is GPU expensive, and not every post-processing stage requires this level of precision. So, we also have a coarse-quality mask reconstruction routine which reduces the reconstruction cost to single texture fetch, from 2 or 4 texture fetches in the regular reconstruction. The coarse quality reconstruction is currently used in Bloom, Depth of field, Down-sampling, Velocity flattening, and Screen-space reflection.
MBFR reduces GPU pixel shading cost by dropping a subset of the pixels in the world rendering passes. But it also introduces extra cost in the post-processing passes for reconstructing the dropped pixels.
By testing on a 4.65ms scene of Robo Recall (on GTX1080), it reduced the overall GPU cost to 4.25ms (-9%). Specifically, it reduced the world rendering (BasePass) cost from 2.51ms to 2.19ms (-13%); and the PostProcessing cost was increased from 0.47ms to 0.65ms (+0.18ms).
Since the cost of PostProcessing is relatively fixed in a game, and the world rendering is dynamic, we can expect more than 10% of GPU performance savings from a scene with heavy world objects rendering.
Of course, the overall GPU performance is also heavily dependent on the FOVs of the density rings. An effective way to further optimize the GPU cost from MBFR is to configure them according to the content being rendered in each map. We can apply bigger density rings in maps which contains high frequency objects/textures to preserve the visual quality (like the cityscape maps in Robo Recall that were tested) and reduce the size of the density rings in other maps, to obtain the greatest performance benefit.