WOC: A Handy Webcam-based 3D Online Chatroom

To meet the need for online real-time chat, we need to simultaneously estimate the 3D motion states (Liu et al., 2022) of all people presented in the webcams. Built on the real-time multi-person motion capture method, ROMP (Sun et al., 2021), we estimate 3D motion in three steps.

3D Motion Capture from a Single Image. Given a single RGB image, ROMP estimates the 3D motion state of each person, represented as 3D SMPL (Loper et al., 2015) pose parameters and 3D translation $T\in R^3$ in camera space. However, ROMP was trained to estimate from a single image. And its video predictions suffer from temporal jittering. To extract the smooth 3D body motion sequence of all the people presented in the webcam, we further need to associate the predictions via tracking and motion smoothing.

3D-detection-based Tracking. Based on the popular 2D tracker ByteTrack (Zhang et al., 2021; Liu et al., 2018), we develop a 3D tracking algorithm 3D-Tracker, which takes the distinguishable 3D translation of ROMP as input to replace the original 2D bounding box input. In this way, we do not need to involve an additional 2D detector to perform tracking, which greatly alleviates the computational burden on our real-time system. More specifically, we convert ROMP’s predicted 3D translation $T=(x,y,z)$ to $(u,v,z,s)$ where $(u,v)$ are the 2D image coordinates obtained via projecting 3D position to 2D image plane, $s$ is the inverse of $z$ to present the person scale in image. Therefore, the system can associate the single-frame predictions to form the 3D body motion sequence of each person.

Motion Smoothing. To filter out the high-frequency jittery in the predicted motion sequence, we develop a motion smoothing sub-module. This module is built on the well-known OneEuro filter (Casiez et al., 2012). For each 3D body motion sequence, we create three individual OneEuro filters to separately process the estimated body 3D orientations, 3D poses, and 3D translations, and then obtain the stable and accurate 3D body motion sequence of all people presented in the webcam frames for avatar animation.

To animate the 3D avatars uploaded by users with estimated SMPL pose parameters, we develop an auto-rigging blender addon¹¹1https://github.com/yanch2116/CDBA to generate and bind compatible SMPL skeletons to 3D avatars.

3D SMPL Skeleton Generation. Given a new uploaded 3D avatar, we first normalize the scale of its mesh and align it to the origin. Then we employ a MeshCNN from NBS (Li et al., 2021) to extract the geometric and topological features from the normalized mesh and estimate the 3D position of 24 joints defined by SMPL.

Automatic Rigging. To bind the generated 3D skeleton to the avatar, we employ an auto-rigging algorithm embedded in Blender (Community, 2018) to calculate the skinning weights of each joint. In this way, we can convert the joint 3D rotation, represented by estimated SMPL pose parameters, into the motion of the mesh vertices. Besides, for Maximo²²2https://www.mixamo.com/ users, we also develop a module to convert the avatar to our format for further animation.

To alleviate the computational burden on users’ clients and enable flexible usage, WOC is deployed in a distributed architecture. On user client, with permission from users, video and audio are collected from their webcams and microphones. To send the data stream to the cloud server, the system establishes peer-to-peer communication via WebRTC. From the cloud server, the system receives the motion sequences of all people for animating avatars. We employ the three.js to show the generated 3D avatar animations on the webpage.

The cloud server collects the media streaming and the uploaded new avatars from all user clients. For each user client, we launch an individual process of motion capture module to estimate the 3D body motion from collected video frames. Then the synchronized results are sent back to all users within the same chatroom. In this way, every user would be able to chat with each other in the chatroom, as shown in Fig. 1.