UE4 and Native Iteration Time Improvements

Oculus Developer Blog
|
Posted by Joe Kirchoff & Remi Palandri
|
July 7, 2020
|
Share

Developer iteration time is one of the top pain points we hear from Oculus developers, and we recently published improvements we made for Unity iteration time. We have also been working hard to reduce iteration time for UE4 and native developers and would like to share some of our latest improvements to the latest Oculus GitHub release.

Skip APK build when iterating on code

Running gradle to generate an updated APK can be a time-consuming process. When iterating on code changes, this means the user needs to wait for their code to compile and link, and then also wait for the APK to be packaged. This improvement allows a user to bypass repackaging an APK when iterating on code changes.

How it works

Instead of requiring the APK be rebuilt to include newer compiled binaries, a change was made to the Oculus OS to allow libraries to load automatically from the application's dataDir instead of from the installed package if a file by the name exists, when a special flag is set in the package settings and the application is debuggable.

Support for UE4

You can currently find this option located in Editor Preferences > General > Experimental.

You can also find this option in the Launch dropdown menu whenever Oculus devices are enabled in the Android project settings.

Notes

This option bypasses the normal Android APK build logic. If any changes are made that require a new APK to be generated, such as Java changes or Android manifest changes, you will need to disable this option temporarily to allow it to be built and installed.

Support for Native Development

Requirements

  • Meta-data needs to be added to the Android manifest to allow the application to load libraries from the dataDir.

meta-data android:name="com.oculus.extlib" android:value="true"

  • Additionally, the installed APK package must also be debuggable.

This is for development purposes only. No application will be accepted on the Oculus Store with this flag enabled.

Usage Instructions

If your APK contains a native library called libName that you wish to modify without repackaging and reinstalling the whole APK, the steps are:

  1. Build an APK with the above requirements met and install it

  2. Compile code and link binaries

  3. Optionally strip debug symbols from binaries to reduce file size (reduces time to push to device): llvm-strip --strip-debug -o lib-stripped.so lib.so

  4. Stop the running app: adb shell am force-stop [package_name]

  5. Push lib to device: adb push lib.so /sdcard/Oculus/Temp/lib.so

  6. Copy to lib load path: adb shell run-as [package_name] cp /sdcard/Oculus/Temp/lib.so ./lib.so

  7. Set executable flag: adb shell run-as [package_name] chmod +x ./lib.so

  8. Start app

Please be aware that any libraries in dataDir will need to be deleted before installing a new APK when you are done iterating, or an older library could be loaded from the dataDir instead of loading from the package. This can be done automatically by first uninstalling the package with adb uninstall [package_name] or by manually deleting any libraries with adb shell run-as [package_name] rm *.so

Results

Testing was done using the VR Template provided with UE4. On average, bypassing the APK build when unneeded saved 129 seconds for a 2.95x improvement in time to Launch.

FASTBuild support for UE4 code compilation

We added support for distributed code compiles using FASTBuild, an open source build system that facilitates distributed work similar to Incredibuild, distcc, or SN-DBS. FASTBuild can help to greatly speed up large code compiles when remote compute resources can be utilized.

Distributed compiles with FASTBuild are currently supported for both MSVC and clang toolchains, so you can use it to compile both Unreal Editor and any Oculus VR game projects.

Requirements

  • Additional Windows machines to do remote work, accessible via port 31264.

  • The Windows machines should either be on your local network or available via VPN, as the network transfer for FASTBuild is not encrypted and should not be sent over the unsecured internet.

Please be aware that the FASTBuild port should not be exposed to the public internet as anyone who can connect to it could run arbitrary commands which could pose a security risk.

Usage Instructions

  • Download FASTBuild and extract the binaries to a folder in %PATH% on every machine.

  • On the remote windows machines, run FBuildWorker.exe. You may be prompted to allow it through the Windows firewall. You can adjust the idle settings and the number of cpu cores to use after opening.

  • Set up worker discovery via the FASTBUILD_BROKERAGE_PATH environment variable as described in the FASTBuild Network Distribution documentation.

  • If a network share is not available, you can instead hardcode remote workers hostnames or ip addresses in the FASTBUILD_WORKERS environment variable, separated by semicolons.

  • UE4 priorities Incredibuild over FASTBuild if both are available, so you may need to uninstall it for FASTBuild to be used.

  • Troubleshooting documentation is also available for FASTBuild if distribution is not working correctly.

  • You can also use another open source project FASTBuild Dashboard to visualize a build if desired.

Code compiles for supported platforms should now be able to generate the FASTBuild config file and distribute tasks to remote workers.

Implementation Details

The implementation was based on the open source project Unreal_FASTBuild and modified to work for newer MSVC toolchains and to support clang.

Unreal compile actions are now processed to determine the correct compiler toolchain to generate for FASTBuild, and to rewrite any command line parameters to match the FASTBuild requirements. Any other non-compile action is run on the local machine, such as generating precompiled headers, linking libraries, or generating final binaries. These actions were complex to generically support converting to the FASTBuild format and didn’t benefit from distribution or caching. As FASTBuild doesn’t support multiple outputs from an action compile dependency information can only be gathered after the build process is complete from the FASTBuild report file, which means currently if a build is cancelled mid process those results will not be saved.

Additionally FASTBuild has built-in caching support for code compiles where object files can be retrieved from the cache if all inputs match both full file path and checksum. Please also be aware that there is some overhead to a FASTBuild compile due to preprocessing all compilation units to gather input files, so it is not recommended to enable unless at least one remote worker is available.

Results

Tests were run on Intel® Xeon® W-2135 Processor [3.70Ghz 12 threads] of a full clean rebuild of the UE4 editor project and of the ShowdownVR sample. These timings include all preprocessing, precompiled header generation, code compiling, and linking.