TL;DR: We are introducing a new technique to optimize VR rendering by reprojecting rendered pixels from one eye to the other eye, and then filling in the gaps with an additional rendering pass. In our Unity sample, this idea has proven to work very well for pixel shader heavy scenarios, saving up to 20+% GPU cost per frame for our test scene. It is also very easy to integrate into your own Unity projects. The sample is available for direct download on the Unity Asset Store.
Typical virtual reality apps render the scene twice, once from the left eye’s view, and once from the right eye’s view. The two rendered images usually look very similar. Intuitively, one would think that maybe we can share some pixel rendering work between both eyes, so we implemented a tech called Stereo Shading Reprojection to make pixel sharing possible. Below we’ll provide an overview of this solution, different scenarios for optimization, and integration best practices.
Basic Theory
The basic idea is to avoid rendering pixels twice and save GPU costs by using the information from the depth buffer to reproject the first eye’s rendering result into the second eye’s framebuffer. While this basic solution seems easy to implement, for delivering a comfortable VR experience we have to make sure:
And, in order to make this technology practical, it should:
Let’s take a look at how reprojection works. Here is the basic procedure (in a typical forward renderer):
There are also a few additional important components to highlight.
Reprojection
Instead of reprojecting from the first eye to the second eye directly, we actually handle the reprojection backwards.
First, a fullscreen quad is drawn with the reprojection shader. Inside the shader, the right (second) eye’s depth buffer is used to reconstruct each pixel’s world-space position, which can be done easily by using the right eye’s inverse view projection matrix, then project the pixel back into left eye’s frame buffer by using left eye camera view projection matrix. Finally, the corresponding color pixel is pulled and output to the frame buffer.
See the images below. At the end of this reprojection process, the color frame buffer would look something like the image on the left. this image appears correct, but looking a bit closer, there are still visual artifacts on the object's edge. The right image shows these artifacts in green. In the next section we'll discuss how to fill these holes.
Occluded Area Detection
Due to something called “binocular parallax,” our left and right eyes see objects from slightly different positions, which results in some pixels the right eye can see that are simply not available in the left eye’s framebuffer. These pixels are occluded from the left eye’s point of view, which is what causes the edge artifacts. To fix these artifacts, the problematic areas can be re-drawn with correct pixels from the normal right eye camera rendering.
To do this, a reprojected pixel needs to be identified as either a valid-reprojection or a false-reprojection. By masking out the valid-reprojection areas, we know which part the reprojection works well and which part needs to be re-rendered.
In the diagram below, the right camera’s pixel O is occluded in the left camera - if you reproject O back into left eye’s frame buffer, what was stored is actually pixel X. Pixel O and X appear to have different depth values, but as a valid-reprojection pixel they should have the same depth value due to parallel view directions of the two cameras (e.g., pixel A). A pixel is a valid reprojection if the left and right pixel’s depths are approximately equal.
Pixel Culling Mask
Now that we can identify which parts of the image are valid-reprojections, we can mask out these areas to avoid re-rendering them. Most materials work well with reprojection. Mirror or very shiny materials, however, can look wrong since their appearance is very view-dependent. To solve this, our solution gives content creators the ability to disable reprojection on a per-material basis.
Either the depth test or stencil test can be used for per-pixel culling, the choice depends mainly on the chosen game engine’s architecture.
Note that since we output per pixel depth in the pixel shader to restore the depth buffer this can break the early-z culling optimization on PC hardware. It’s important to do this after the opaque pass is done, so you can minimize the penalty impact and your opaque pass can still benefit from the reprojection.
Conservative Reprojection Filter
If you’ve gotten this far, the reprojection should work, however, there is one more visual artifact that we can’t ignore -- edge ghosting. The left side of following picture shows what edge ghosting looks like This can be fixed by using a method called Conservative Reprojection Filter, which results in the improvement you see at right.
When occluded areas are detected, it’s recommended to check if the reprojected depth matches the source depth. Due to reprojection filtering and floating-point errors, a threshold test instead of an equality test is used. This can introduce artifacts where a reprojected pixel is close to the edge of a foreground object. Since the reprojected position can be anywhere between the foreground pixel and the background pixel, simply reducing the threshold won’t work, and it will eventually disable the reprojection completely.
However, It is easy to detect the foreground edge case and mask those areas as false-reprojection, even if it is valid-reprojection. Basically, make the culling more conservatively. This can be achieved by using a triangle shape filter to get 3 depth samples as below diagram, and only select the closest depth value for depth comparing. This conservative approach won’t hurt the performance gain noticeably since those edges don’t cover a big screen area. In term of the triangle’s size, 2 pixel distance from vertices to center works well in our project.
Performance Results
To put this all to the test, we implemented the stereo shading reprojection idea in Unity through some simple command buffers and shaders, which worked very well for pixel-shader heavy scenes. In our test scene, we intentionally exaggerated the pixel cost by adding multiple dynamic lights. Here is some performance data for the following scene running on GTX 970:
We can see the opaque pass cost drops from 4.6ms to 3.4ms, which is about a ~26% saving, this included all the overhead introduced by the reprojection ( about 0.7ms for this case ). In total, the whole frame GPU cost drops from 5.6ms to 4.4ms -- still a ~21% saving. Depending on the chosen MSAA level and framebuffer format , the reprojection overhead can vary due to the cost of the MSAA resolve ( 0.5ms - 1.2ms ). Because reprojection overhead is a constant cost, the more expensive opaque shading pass resulted in a better saving percentage. We also profiled it on AMD R290 and GTX 1080 and observed similar savings. We expect the constant reprojection overhead becomes more trivial when the GPU becomes faster.
Limitations
Stereo reprojection may not work for all cases, and has some limitations:
Notes on Unity Integration
To integrate the Unity implementation process into your project simply attach the script StereoReprojectionPass.cs under your main camera object. If there are highly specular materials that need to avoid reprojecting, modify the shader to output alpha = 0. An n example of this is provided in the Unity implementation project for reference.
Notes:
Below is a simple diagram which illustrates the whole pipeline in Unity: