How does Oculus Link Work? The Architecture, Pipeline and AADT Explained

Connecting a USB cable between a compatible PC and Oculus Quest headset enables Oculus Link, unlocking PC VR content for Oculus Quest owners without additional development costs from developers.

Oculus Link is a unique intersection between PC and mobile Runtimes. In this article we will share the challenges and solutions designed to bring these VR Runtimes together to offer a high quality, low latency VR experience over a USB cable. We hope you find this article both insightful and interesting.

Getting the lowest possible latency in VR requires an end-to-end understanding of everything that’s going to happen in the system -- if latency in the pipeline causes the virtual world to lag behind the player, the result is a bad experience. If, however, latency can be anticipated, the VR experience doesn’t have to lag at all -- instead, the VR Runtime can compensate, mitigating specific artifacts that make such small amounts of latency so obvious to our senses.

At the heart of many VR optimizations is a very specific understanding of how every piece of the system is going to work, from the operating system’s scheduler, to the display’s illumination pattern and the mechanics of human perception. To get the maximum performance out of streaming, and to leverage the optimizations that are built into the PC and Oculus Quest Runtimes, Oculus Link needed to have a fully coherent VR pipeline, beginning with the application on the PC and ending with display illumination on Oculus Quest.

At the highest level, Oculus Link works by streaming inputs and outputs between the Oculus PC Runtime and a client application running on the Oculus Quest.

There are no changes to the PC app, or to the compositor running on Oculus Quest. Rather than rendering a backbuffer for the headset display, the PC compositor encodes the eye textures -- surfaces that an application submits to the compositor -- using H.264 video compression, and sends the images as slices to the remote client over USB.

On the Oculus Quest, the remote streaming client is responsible for decoding, rectifying and submitting the frame to the compositor, acting as a proxy for the PC application on the remote HMD. The client also sends updates to the PC Runtime with the current HMD position, input state and VSync timings. These are sent without prediction - the PC VR Runtime extrapolates these values to match the target display period.

Effectively, the VR compositor’s responsibilities are split between the mobile VR and PC VR Runtimes, turning the ordinary VR pipeline into a hybrid. The split has two benefits:

1. Oculus Link is able to leverage the VR Runtime on both sides — rather than incur a blanket penalty for transmitting inputs and outputs over USB, the pipeline has correction at the end of submission that will mitigate some of the transmission latency.

2. If done right, existing Rift and Rift S applications continue to operate without any knowledge of the streaming backend.

An important responsibility of the VR Runtime is to accurately predict the pipeline’s latency so that the application renders its frames for the target display period. Another important responsibility is to make sure that the frame is never drawn for the wrong display period — if it is shown too early or too late, the virtual world judders and lags in the eyes of the player.

From this perspective, a hybrid pipeline doesn’t work out-of-box — when the two Runtimes are used together, neither is aware of the additional pipeline stages that were added by streaming.

Our solution is a new frame paradigm in which the PC and mobile Runtimes map to one another: from the perspective of the PC Runtime, the streaming latency of USB and the unknown path on Oculus Quest become longer compositor stage; and from the perspective of the mobile Runtime, the unknown path from app render to frame submission looks like a dynamic application.

The predicted display time is an important way the Oculus PC Runtime gets the lowest possible latency out of applications. If applications could not assume frames were being prepared for a particular display period, they would need to buffer several frames ahead to account for the unknown pipeline stages that occur after application frame submission.

Unlike on the Oculus Quest, PC VR applications cannot assume a fixed hardware configuration — the run length of the pipeline will always vary depending on what CPU and GPU the user has in their PC. Therefore, the Runtime will create its estimate based on the historical averages of four pipeline stages: the application’s frame loop (CPU and GPU) compositor time, and scanning out to the display.

For Oculus Link, the Rift PC pipeline estimate falls short. However, the idea can be extended to account for the additional stages that were added by streaming. When a frame is displayed on the Oculus Quest, the streaming client sends the end timestamp back to the PC Runtime; the streaming-pipeline-end timestamp is then substituted for the compositor-end timestamp in the VSync calculation; because the PC Runtime is already equipped to handle variable compositor times, it dynamically adjusts its prediction to match the end-to-end-latency of streaming without having to reduce framerate.

Normally, the Oculus Quest application receives a pose, renders to that pose, then submits for the target display period. Oculus Link is different in that the predicted frames arrive from a remote device. When a frame’s predicted display time does not align with its actual display time, the result is a misprediction. Mispredictions by a single frame or two are often not visible; however, oscillation between predictions creates judder.

To fix this, Oculus Link leverages the PC Runtime’s Asynchronous Timewarp (ATW) and Adaptive Compositor Kickoff (ACK). ATW guarantees that a frame is always ready for the upcoming display period. ACK makes sure that ATW kicks off with exactly the amount of time needed to hit its deadline. See Volga’s OC5 presentation below for a more in-depth overview of ACK:

For streaming, ACK can be used to make sure that a frame targeting the correct display period is ready to be submitted for every frame interval on the Oculus Quest. When streaming latency is added to ACK’s total estimate of the display pipeline, ATW makes sure to kick off a frame just in time so that it can be decoded, rectified and presented for the upcoming display period.

Delivering high resolution images from PC to Oculus Quest is challenging because the video codecs’ (encoder and decoder) overall latency is directly related to the amount of bits that need to be coded. We could potentially reduce the image resolution to decrease latency, but the lower resolution image will impair visual quality and hinder the experience. We can do better by utilizing the HMD’s lens distortion, along with our understanding of how the human visual system works.

Every Oculus headset that has shipped to date applies distortion correction at the final stage of rendering, before displaying the image. This is to counteract the optical-pincushion distortion caused by the lens. As a result, barrel distortion “squeezes” down the VR app’s pixels rendered closer to the HMD display border. This means, for each half of the display, the pixels-per-degree is lower on the periphery than the center of the display.

The figure above shows a sample radial-barrel distortion applied for final viewing on an HMD display.

The final pixel density is tied to the distortion curvature applied by the GPU. For a given pair of lenses, applying this correction is essential for an accurate VR experience. However, for Oculus Link, the image generated by the PC GPU does not have to reflect the exact distortion curvature required by the Oculus Quest HMD lenses because, unlike the Oculus Rift, the Oculus Quest has its own GPU that allows further processing of incoming images. As a result, the Oculus PC Runtime has room to optimize images before transmission. One major optimization comes from the fact that the lenses slightly blur the peripheral regions, decreasing the perceptual resolution.

This is where Axis-Aligned Distortion Transmission (AADT) comes into play. Instead of directly sending undistorted (i.e. rectilinear) images rendered by the VR app, the Oculus PC Runtime will use axis-aligned distortion to apply a form of fixed-foveated compression. While radial distortion used for lens correction applies a distortion function based on the distance from the center of the image regardless of 2D direction, AADT applies a distortion function to each axis independently. So AADT makes use of all available real-estate in the rectangular image while radial distortion can show black pull-in on the corners leading to wasted encode-decode resources. Once the AADT image is received by the Oculus Quest, it will undistort (i.e. rectify) the image back to its original shape and size before passing it into the Oculus Quest compositor.

With AADT, a pair of undistorted eye-buffers rendered at 3616x2000 can be compressed down to 2016x1120 saving close to 70% of the video codec budget. AADT not only saves codec bandwidth, but it also saves latency by making sure the encoder and decoder work on smaller images. While these numbers are tuned for the Oculus recommended-specification hardware, they might be updated for better visual fidelity in future releases.

The human visual system is particularly good at picking up flickering in the periphery of the FOV. If Oculus Link directly sent undistorted images to the Oculus Quest, undersampling (or sparse sampling) on the Oculus Quest GPU would cause the periphery to be aliased. With AADT, while the Oculus PC compositor is preparing the image to be transmitted, it makes use of anisotropic filtering with mipmaps. This eliminates undersampling issues and reduces the peripheral artifacts after rectification on the Oculus Quest.

Let’s see AADT in action. Observe the grid overlaid on top of the first example followed by a couple of sample screenshots from Oculus Dreamdeck application. After AADT, the undistorted central region is magnified. After rectification on the Oculus Quest, the periphery is slightly blurred while the quality of the central region is preserved.

In this article we described engineering solutions that enable Oculus Link to achieve low latency, accurate timing and high-quality visuals. We saw how PC and Quest frame composition frameworks are set up to work together for better frame estimates. We also surveyed how headset lenses and human vision work, showing how we were able to optimize bandwidth, while maintaining visual fidelity.

As a reminder, you can always monitor the active PC VR app’s performance statistics including app and PC compositor latencies using the Oculus Performance HUD. This tool will work out-of-the-box for Oculus Quest when Oculus Link is enabled.

Please refer to the Oculus Link OC6 presentation below for more technical information and the Oculus Link Compatibility Page to learn more about how to use Oculus Link.

We hope Oculus Link is a great experience for you and your consumer audience, and we look forward to building even more innovative, impactful technologies in the future.

- The Oculus Link Team