17/08/2020

MEgATrack: Monochrome egocentric articulated hand-tracking for virtual reality

Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, Robert Wang

Keywords: hand tracking, virtual reality, motion capture

Abstract Paper Similar Papers

Abstract: We present a system for real-time hand-tracking to drive virtual and augmented reality (VR/AR) experiences. Using four fisheye monochrome cameras, our system generates accurate and low-jitter 3D hand motion across a large working volume for a diverse set of users. We achieve this by proposing neural network architectures for detecting hands and estimating hand keypoint locations. Our hand detection network robustly handles a variety of real world environments. The keypoint estimation network leverages tracking history to produce spatially and temporally consistent poses. We design scalable, semi-automated mechanisms to collect a large and diverse set of ground truth data using a combination of manual annotation and automated tracking. Additionally, we introduce a detection-by-tracking method that increases smoothness while reducing the computational cost; the optimized system runs at 60Hz on PC and 30Hz on a mobile processor. Together, these contributions yield a practical system for capturing a user’s hands and is the default feature on the Oculus Quest VR headset powering input and social presence.

The video of this talk cannot be embedded. You can watch it here:
https://dl.acm.org/doi/10.1145/3386569.3392452#sec-supp
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at SIGGRAPH 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 2:30 
 0
 0
 0
 0
 2:50 
 0
 0
 0
 0
 8:21 
 0
 0
 0
 0
 9:15 
 0
 0
 0
 0
 3:07 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 2:20 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 5:03 
 0
 0
 0
 0
 12:21 
 0
 0
 0
 0
 6:40 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 9:36 
 0
 0
 0
 0
 4:36 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 4:53 
 0
 0
 0
 0
 0:56 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 18:14 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 4:52 
 0
 0
 0
 0
 11:21 
 0
 0
 0
 0
 12:52 
 0
 0
 0
 0
 5:05 
 0
 0
 0
 0
 4:47 
 0
 0
 0
 0
 1:03 
 0
 0
 0
 0
 0:59 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 19:42 
 0
 0
 0
 0
 5:04 
 0
 0
 0
 0
 18:00 
 0
 0
 0
 0
 4:41 
 0
 0
 0
 0
 12:56 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 8:57