Insights from Tribe XR: Social VR + Avatar SDK Basics

Oculus Developer Blog
Posted by Tom Impallomen
November 22, 2019

Today we hear from Tribe XR CEO and co-founder Tom Impallomeni as he gives a number of best practices and insights from building + launching his DJ learning / performance application Tribe XR. He covers a number of social-focused topics, including the importance of avatars, spatial VoIP, and how to get started. If you haven't already, be sure to review the Oculus Avatar SDK documentation and downloads page.

Transporting a user into a completely different world and pushing the boundaries of physical space is, undeniably, VR’s ultimate superpower. The other outstanding benefit of VR, however, that is often overlooked, is its ability to bring people, regardless of their location, to hang out, play, learn, and have fun. As Adam Draper at Boost VC likes to say, VR’s killer app is “being together.”

The feeling of belonging and connecting with others has multiple facets and, in VR, it’s having and interacting with one’s avatar.

Why avatars?

The idea of avatars is not new but it’s especially important for the VR crowd. People interacting in VR primarily take the form of avatars for a number of reasons:

  1. Users are required to wear a VR headset, so their faces are not visible to other people. This necessitates the use of avatars to communicate expressively in VR.
  2. Filming a person in VR requires external cameras (as demonstrated by plugins such as LIV, which enable mixed-reality). Front-facing mixed reality looks a bit odd because people wear headsets which hide their eyes and the majority of facial expressions.
  3. Avatars enable users to create a virtual representation of their self, not limited by their actual features. Avatars are a creative canvas for personal representation.

How Oculus Expressive Avatars improved our D school experience

At Tribe XR, we offer immersive training for creative skills, and with our first product, the Tribe XR DJ School, anyone can learn to DJ, mix sets and livestream performances to VR platforms such as VR Chat and non-VR live streaming platforms such as Twitch, Facebook Live and YouTube Live. And Oculus Avatars are “the face(s)” of our product and are an important part of the user experience.

While Oculus Avatars initially had a few limitations, the launch of Expressive Avatars introduced a number of features that made it an essential part of social VR experience:

  • Eyes that follow the viewer create a sense of connection
  • Animated Mouth and Lips enhance the realism of a conversation
  • Facial expressions, such as smiling, grimacing and so on communicate emotions

These additions create a sense of presence, increase the realism of the experience and generally improve the overall social VR experience which, for Tribe users, translates into a better learning experience.

For Tribe, the connection between teacher and student is critical. Any improvements in avatar realism enhance the learning experience and make it more human. It increases the value of lessons and helps us to realize a key goal: to build lasting connections between students and teachers.

A secondary benefit that improves a student’s experience in Tribe is gaze modeling. When a teacher is talking to a student and communicating to the student to manipulate a specific button or slider on a DJ deck, the ability to gaze towards the location of the button improves the efficiency of the lesson. Tribe was already using visual cues to enable learning, for example, a virtual laser pointer. Gaze modeling improves the efficiency of training.

Does this overcome the uncanny valley? Not completely, but the new expressive avatars are still sufficiently abstracted from reality, and, to us, the avatars with eyes are much more appealing and palatable than avatars without them! A cherry on top is an ability for community members to choose to add sunglasses to their avatars.

Leveraging Oculus Avatars to enable a learning platform

A user’s social experience in Tribe does not stop when they pick their appearance through Oculus avatars: that’s, in fact, where the journey starts. In Tribe, users interact through their avatars in a number of ways: by simply having an avatar of choice when mixing in Tribe on their own, when having a lesson with a tutor, when mixing or playing music back-to-back together with their friends, and when performing with a worldwide audience through live streaming.

1-2-1 and group lessons

Tribe’s individual and group lessons can be experienced synchronously or asynchronously. We use Oculus’ Expressive Avatars SDK to enable teachers and students to create life-like avatars so they can communicate in a way that feels realistic and present, and so that each participant of this “exchange” could easily identify themselves and the teacher.

In the screengrab below, one of our resident DJ mentors Ellaskins gives a VR lesson from VR, streamed live on YouTube. Ellaskins is a well-known YouTube DJ Mentor who has built a following of almost 200,000 people since 2006 by creating free DJ lessons and tutorials, covering everything from DJ basics to advanced techniques. Represented by an Oculus Avatar, Ellaskins communicates to his students and his wider YouTube audience through Tribe XR in a fun and engaging way. The avatars, virtual decks and custom virtual camera views make it easy for Ellaskins to convey his techniques, and for students to follow along. The facial animations and expressions help Ellaskins’ avatar feel more real, helping overcome the challenges presented by the uncanny valley. The lessons created in Tribe are useful both to VR and real-life DJs using traditional hardware.

“Back-to-back” multiplayer VR DJing

With Tribe’s multiplayer mode, our community members can collaborate to create music and DJ sets together in a live session. Users can DJ “back-to-back,” taking turns playing music and operating the virtual turntables together. With Oculus avatars, users create their own characters that represent them and reflect their creativity and make the back-to-back DJ sets engaging and fun to watch.

Live streaming performances and lessons

Tribe XR is built with the goal of helping students practice their performances. Many of our DJs learn about live performance through live streaming to Twitch and other platforms. Our teachers also create lesson content for live streaming platforms, Ellaskins being one example of this. Here, Jonn Connor (aka Seventh Sign) teaches the basics of counting beats and bars.

Tribe livestreams on Twitch allow listeners to interact with the DJ through messages, GIFs and by sending emotes. Avatars are often the DJs alter-ego that helps them both realize their creativity and be a memorable performer for the audience.

Technical implementation

Tribe XR is built with Unreal Engine and utilizes the Oculus SDK’s networking and VoIP capabilities to enable multiplayer experiences.

The implementation of Expressive Avatars is straightforward if you are already using the Oculus Avatars SDK, which is cross-platform. To upgrade to Expressive Avatars, simply replace the old avatars version with the new. Much of the work we did on top of the SDK was tuning things like gaze to work well with our environment and the virtual decks, functionality that is provided as part of the SDK.

Tribe uses the UE4 Oculus Avatar plugin with some custom modifications such as making the avatar's body visible to 3rd person cameras but not the player (as it might obstruct the view of the DJ decks). We have a custom component for Pawns that interfaces with the Oculus Avatar plugin, handling spawning avatars and replicating them across the network. We also updated the avatar component to animate the avatar's mouth using Oculus Avatars' lip sync plugin, and added gaze targets to give the avatar's eyes something to focus on.

We implemented Oculus Avatars across other VR platforms as well. We did this by replicating Oculus’ Unity instructions to package the necessary library and assets in UE4. We then modified the avatar plugin to use local library copies to support non-Oculus users who don’t have Oculus Home installed.

Finally, we updated the avatar plugin to make avatars track and animate properly with non-Oculus HMDs and controllers by using UE4's generic HMD interface for head and hand positions, and input data to animate the hands. This way, the avatars support all VR APIs that UE4 supports, including all devices that work with SteamVR. We added a custom menu so users can choose between the 12 default Oculus Expressive Avatars. We plan on offering some special default avatars for Tribe in the near future.

What’s next for Tribe?

Our goal is to continue powering immersive training for creative skills, with multiplayer lessons as a key feature. We are very excited about Oculus Expressive Avatars and the way they enhance group lessons. We also look forward to developing further customization options and enabling livestreamers to create their own DJ brands and personas in VR. Tribe XR DJ School is now available in single and multiplayer modes with cross buy on Oculus Quest and Rift!

More on Tribe XR

To learn more about implementing social interactions in your VR game or application, be sure to check out the following blog post as it covers the latest social features coming to the Platform and Avatar SDKs, including upgraded expressive avatars, spatial VoIP and more. We also discussed this topic at length during OC6, see Mike Howard’s presentation below for more about these products, and best practices for implementing these features in your next game or application.