VrApi

Android Activity lifecycle

An Android Activity can only be in VR mode while the activity is in the resumed state. The following shows how VR mode fits into the Android Activity lifecycle.

     1.  VrActivity::onCreate() <---------+
     2.  VrActivity::onStart()  <-------+ |
     3.  VrActivity::onResume() <---+   | |
     4.  vrapi_EnterVrMode()        |   | |
     5.  vrapi_LeaveVrMode()        |   | |
     6.  VrActivity::onPause() -----+   | |
     7.  VrActivity::onStop() ----------+ |
     8.  VrActivity::onDestroy() ---------+     

Android Surface lifecycle

An Android Activity can only be in VR mode while there is a valid Android Surface. The following shows how VR mode fits into the Android Surface lifecycle.

     1.  VrActivity::surfaceCreated() <----+
     2.  VrActivity::surfaceChanged()      |
     3.  vrapi_EnterVrMode()               |
     4.  vrapi_LeaveVrMode()               |
     5.  VrActivity::surfaceDestroyed() ---+

Note that the lifecycle of a surface is not necessarily tightly coupled with the lifecycle of an activity. These two lifecycles may interleave in complex ways. Usually surfaceCreated() is called after onResume() and surfaceDestroyed() is called between onPause() and onDestroy(). However, this is not guaranteed and, for instance, surfaceDestroyed() may be called after onDestroy() or even before onPause().

An Android Activity is only in the resumed state with a valid Android Surface between surfaceChanged() or onResume(), whichever comes last, and surfaceDestroyed() or onPause(), whichever comes first. In other words, a VR application will typically enter VR mode from surfaceChanged() or onResume(), whichever comes last, and leave VR mode from surfaceDestroyed() or onPause(), whichever comes first.

Android VR lifecycle

This is a high-level overview of the rendering pipeline used by VrApi. For more information, see VrApi\Include\VrApi.h.

1. Initialize the API.
2. Create an EGLContext for the application.
3. Get the suggested resolution to create eye texture swap chains with vrapi_GetSystemPropertyInt( &java, VRAPI_SYS_PROP_SUGGESTED_EYE_TEXTURE_WIDTH ).
4. Allocate a texture swap chain for each eye with the application's EGLContext current.
5. Get the suggested FOV to setup a projection matrix.
6. Setup a projection matrix based on the suggested FOV. Note that this is an infinite projection matrix for the best precision.
7. Android Activity/Surface lifecycle loop.
   1. Acquire ANativeWindow from Android Surface.
   2. Enter VR mode once the Android Activity is in the resumed state with a valid ANativeWindow.
   3. Frame loop, possibly running on another thread.
   4. Get the HMD pose, predicted for the middle of the time period during which the new eye images will be displayed. The number of frames predicted ahead depends on the pipeline depth of the engine and the synthesis rate. The better the prediction, the less black will be pulled in at the edges.
   5. Apply the head-on-a-stick model if there is no positional tracking.
   6. Advance the simulation based on the predicted display time.
   7. Render eye images and setup ovrFrameParms using 'ovrTracking'.
   8. Render to 'textureId' using the 'eyeViewMatrix' and 'eyeProjectionMatrix'. Insert 'fence' using eglCreateSyncKHR.
   9. Hand over the eye images to the time warp.
   10. Must leave VR mode when the Android Activity is paused or the Android Surface is destroyed or changed.
   11. Destroy the texture swap chains. Make sure to delete the swapchains before the application's EGLContext is destroyed.
8. Shut down the API.

Integration

The API is designed to work with an Android Activity using a plain Android SurfaceView, where the Activity lifecycle and the Surface lifecycle are managed completely in native code by sending the lifecycle events (onResume, onPause, surfaceChanged etc.) to native code.

The API does not work with an Android Activity using a GLSurfaceView. The GLSurfaceView class manages the window surface and EGLSurface and the implementation of GLSurfaceView may unbind the EGLSurface before onPause() gets called. As such, there is no way to leave VR mode before the EGLSurface disappears. Another problem with GLSurfaceView is that it creates the EGLContext using eglChooseConfig(). The Android EGL code pushes in multisample flags in eglChooseConfig() if the user has selected the "force 4x MSAA" option in settings. Using a multisampled front buffer is completely wasted for TimeWarp rendering.

Alternately, an Android NativeActivity can be used to avoid manually handling all the lifecycle events. However, it is important to select the EGLConfig manually without using eglChooseConfig() to make sure the front buffer is not multisampled.

The vrapi_GetSystemProperty* functions can be called at any time from any thread. This allows an application to setup its renderer, possibly running on a separate thread, before entering VR mode.

On Android, an application cannot just allocate a new window/frontbuffer and render to it. Android allocates and manages the window/frontbuffer and (after the fact) notifies the application of the state of affairs through lifecycle events (surfaceCreated / surfaceChanged / surfaceDestroyed). The application (or 3rd party engine) typically handles these events. Since the VrApi cannot just allocate a new window/frontbuffer, and the VrApi does not handle the lifecycle events, the VrApi somehow has to take over ownership of the Android surface from the application. To allow this, the application can explicitly pass the EGLDisplay, EGLContext EGLSurface or ANativeWindow to vrapi_EnterVrMode(), where the EGLSurface is the surface created from the ANativeWindow. The EGLContext is used to match the version and config for the context used by the background time warp thread. This EGLContext, and no other context can be current on the EGLSurface.

If, however, the application does not explicitly pass in these objects, then vrapi_EnterVrMode() must be called from a thread with an OpenGL ES context current on the Android window surface. The context of the calling thread is then used to match the version and config for the context used by the background TimeWarp thread. The TimeWarp will also hijack the Android window surface from the context that is current on the calling thread. On return, the context from the calling thread will be current on an invisible pbuffer, because the time warp takes ownership of the Android window surface. Note that this requires the config used by the calling thread to have an EGL_SURFACE_TYPE with EGL_PBUFFER_BIT.

Before getting sensor input, the application also needs to know when the images that are going to be synthesized will be displayed, because the sensor input needs to be predicted ahead for that time. As it turns out, it is not trivial to get an accurate predicted display time. Therefore the calculation of this predicted display time is part of the VrApi. An accurate predicted display time can only really be calculated once the rendering loop is up and running and submitting frames regularly. In other words, before getting sensor input, the application needs an accurate predicted display time, which in return requires the renderer to be up and running. As such, it makes sense that sensor input is not available until vrapi_EnterVrMode() has been called. However, once the application is in VR mode, it can call vrapi_GetPredictedDisplayTime() and vrapi_GetPredictedTracking() at any time from any thread.

The VrApi allows for one frame of overlap which is essential on tiled mobile GPUs. Because there is one frame of overlap, the eye images have typically not completed rendering by the time they are submitted to vrapi_SubmitFrame(). To allow the time warp to check whether the eye images have completed rendering, the application can explicitly pass in a sync object (CompletionFence) for each eye image through vrapi_SubmitFrame(). Note that these sync objects must be EGLSyncKHR because the VrApi still supports OpenGL ES 2.0.

If, however, the application does not explicitly pass in sync objects, then vrapi_SubmitFrame() must be called from the thread with the OpenGL ES context that was used for rendering, which allows vrapi_SubmitFrame() to add a sync object to the current context and check if rendering has completed.

Note that even if no OpenGL ES objects are explicitly passed through the VrApi, then vrapi_EnterVrMode() and vrapi_SubmitFrame() can still be called from different threads. vrapi_EnterVrMode() needs to be called from a thread with an OpenGL ES context that is current on the Android window surface. This does not need to be the same context that is also used for rendering. vrapi_SubmitFrame() needs to be called from the thread with the OpenGL ES context that was used to render the eye images. If this is a different context than the context used to enter VR mode, then for stereoscopic rendering this context *never* needs to be current on the Android window surface.

Eye images are passed to vrapi_SubmitFrame() as "texture swap chains" (ovrTextureSwapChain). These texture swap chains are allocated through vrapi_CreateTextureSwapChain(). This is important to allow these textures to be allocated in special system memory. When using a static eye image, the texture swap chain does not need to be buffered and the chain only needs to hold a single texture. When the eye images are dynamically updated, the texture swap chain needs to be buffered. When the texture swap chain is passed to vrapi_SubmitFrame(), the application also passes in the chain index to the most recently updated texture.

The VrApi Input API allows applications linked to VrApi to enumerate and query the state of devices connected to a Gear VR. Once a device is enumerated, its current state can be queried each frame using the input API.

The Input API is defined in VrApi/Src/VrApi_Input.h. For sample usage, see VrSamples/Native/VrController.

For a discussion of best practices, see Gear VR Controller Best Practices in Oculus Best Practices.

Input Devices

Input API supports the Gear VR Headset and Gear VR Controller. Gear VR Headset controls include the touchpad and Back button short-press.

The Gear VR Controller is an orientation-tracked input device. Gear VR positions the controller relative to the user by using a body model to estimate the controller’s position. Left-handedness versus right-handedness is specified by users during controller pairing and is used to visualize the controller on the appropriate side of the user’s body in VR.

The Home, Back button long-press, volume up and volume down buttons on the controller and headset are reserved for system use and will not appear in the button state on either input device.

Enumerating Devices

Enumerated devices may be the Gear VR Headset or the Gear VR Controller.

In order to find a device, an application should call vrapi_EnumerateInputDevices. This function takes a pointer to the ovrMobile context, and an index and a pointer to an ovrInputCapabilityHeader structure. If a device exists for the specified index, the ovrInputCapabilityHeader’s Type and DeviceID members are set upon return.

Once a device is enumerated, its full capabilities can be queried with vrapi_GetInputDeviceCapabilities. This function also takes a pointer to an ovrInputCapabilityHeader structure, but the caller must pass a structure that is appropriate for the ovrControllerType that was returned by vrapi_EnumerateInputDevices.

For instance, if vrapi_EnumerateInputDevices returns a Type of ovrControllerType_TrackedRemote when passed an index of 0, then the call to vrapi_GetInputDeviceCapabilities should pass a pointer to the Header field inside of a ovrInputTrackedRemoteCapabilities structure. For example:

ovrInputCapabilityHeader capsHeader;
if ( vrapi_EnumerateInputDevices( ovrContext, 0, &capsHeader ) >= 0 )
{
   if ( capsHeader.Type == ovrControllerType_TrackedRemote )
   {
      ovrInputTrackedRemoteCapabilities remoteCaps;
      if ( vrapi_GetInputDeviceCapabilities( ovr, &remoteCaps.Header ) >= 0 )
      {
            // remote is connected
      }
   }
}

After successful enumeration, the ovrInputCapabilityHeader structure that was passed to vrapi_EnumerateInputDevices will have its DeviceID field set to the device ID of the enumerated controller.

From this point on, the device state can be queried by calling vrapi_GetInputTrackingState as described below.

Device Connection and Disconnection

Devices are considered connected once they are enumerated through vrapi_EnumerateInputDevices, and when vrapi_GetInputTrackingState and vrapi_GetCurrentInputState return valid results.

vrapi_EnumerateInputDevices does not do any significant work and may be called each frame to check if a device is present or not.

Querying Device Input State

The state of the controller and Gear VR headset can be queried via the vrapi_GetCurrentInputState function.

Both functions take deviceIDs and pointers to ovrInputStateHeader structures. Before calling these functions, fill in the header’s Type field with the type of device that is associated with the passed deviceID. Make sure the structure passed to these functions is not just a header, but the appropriate structure for the device type. For instance, when querying a controller, pass an ovrInputTrackedRemoteCapabilities structure with the Header.Type field set to ovrControllerType_TrackedRemote.

ovrInputStateTrackedRemote remoteState;
remoteState.Header.Type = ovrControllerType_TrackedRemote;
if ( vrapi_GetCurrentInputState( ovr, controllerDeviceID, &remoteState.Header ) >= 0 )
{
// act on device state returned in remoteState
}

vrapi_GetCurrentInputState returns the controller’s current button and trackpad state.

Querying Device Tracking State

To query the orientation tracking state of a device, call vrapi_GetInputTrackingState and pass it a predicted pose time. Passing a predicted pose time of 0 will return the most recently-sampled pose.

ovrTracking trackingState;
if ( vrapi_GetInputTrackingState( ovr, controllerDeviceID, &trackingState ) >= 0 )

VrApi implements an arm model that uses the controller’s orientation to synthesize a plausible hand position each frame. The tracking state will return this position in the Position field of the predicted tracking state’s HeadPose.Pose member.

Controller handedness may be queried using vrapi_GetInputDeviceCapabilities as described in Enumerating Devices above.

Applications that implement their own arm models are free to ignore this position and calculate a position based on the Orientation field that is returned in the predicted tracking state’s pose.

Recentering the Controller

Users may experience some orientation drift in the yaw axis, causing the physical controller's orientation to go out of alignment with its VR representation.

To synchronize the physical controller’s orientation with the VR representation, users should:

1. Point the controller in the direction of the forward axis of their headset, and
2. Press and hold the Home button for one second.

When a recenter occurs, the VrApi arm model is notified and the arm model’s shoulders are repositioned to align to the headset’s forward vector. This is necessary because the shoulders do not automatically rotate with the head.

Applications that implement their own arm models can poll the device input state’s RecenterCount field to determine when the controller is recentered. RecenterCount increments only when a recenter is performed. We recommend recentering arm models based on the head pose when this field changes.

Headset Emulation

Emulation mode is convenient for applications that have not been rebuilt to use the new controller API. When enabled, Gear VR Controller touch values use the same mapping as headset touch values, and applications cannot distinguish headset inputs from controller inputs.

Headset emulation for the controller can be toggled on or off by calling vrapi_SetRemoteEmulation. It is enabled by default.

When emulation is enabled, applications that load a new VrApi with Gear VR Controller support will receive input from the controller through Android Activity’s dispatchKeyEventx and dispatchTouchEvent methods.

New applications and applications that are specifically updated to use the controller should use the VrApi Input API to enumerate the controller and query its state directly. Applications may also want to enumerate the headset and query its state through the same API.

Gear VR Controller Swiping Gestures

For Gear VR Controllers, the user interface of your VR experience should follow these natural scrolling and swiping gestures:

• Swipe up: Pull content upward. Equivalent to scrolling down.
• Swipe down: Pull content downward. Equivalent to scrolling up.
• Swipe left: Pull content left or go to the next item or page.
• Swipe right: Pull content right or go to the previous item or page.