Cornell researchers used a tiny camera and a specially designed deep neural network to create a first-of-its-kind wristband that records the entire body position in 3D. BodyTrak is the primary wearable that uses a single camera to trace the entire body pose. BodyTrak, if integrated into future smartwatches, could be a game changer in monitoring consumer body mechanics in bodily activities where precision is critical, according to Cheng Zhang, assistant professor of data science and the paper’s senior author.
Cornell researchers created a first-of-its-kind wristband that tracks entire body posture in 3D using a miniature camera and a customized deep neural network. BodyTrak is the first wearable that uses a single camera to track the full body pose. BodyTrak, if integrated into future smartwatches, could be a game changer in monitoring user body mechanics in physical activities where precision is critical, according to Cheng Zhang, assistant professor of information science and senior author of the paper.
Our research shows that we don’t need our body frames to be fully within camera view for body sensing. If we are able to capture just a part of our bodies, that is a lot of information to infer to reconstruct the full body.
Hyunchul Lim
“Since smartwatches already have a camera, technology like BodyTrak could understand the user’s pose and give real-time feedback,” Zhang said. “That’s handy, affordable and does not limit the user’s moving area.”
A corresponding paper, “BodyTrak: Inferring Full-body Poses from Body Silhouettes Using a Miniature Camera on a Wristband,” was published in the Proceedings of the Association for Computing Machinery (ACM) on Interactive, Mobile, Wearable and Ubiquitous Technology, and presented in September at UbiComp 2022, the ACM international conference on pervasive and ubiquitous computing.
BodyTrak is the latest body-sensing system from the SciFiLab, which is based in Cornell’s Ann S. Bowers College of Computing and Information Science and has previously developed and leveraged similar deep learning models to track hand and finger movements, facial expressions, and even silent-speech recognition.
The secret to BodyTrak is not only the dime-sized camera on the wrist, but also the customized deep neural network that powers it. This deep neural network reads the camera’s rudimentary images or “silhouettes” of the user’s body in motion and virtually re-creates 14 body poses in 3D and real time.
In other words, the model accurately fills out and completes the partial images captured by the camera, said Hyunchul Lim, a doctoral student in the field of information science and the paper’s lead author.
“Our research shows that we don’t need our body frames to be fully within camera view for body sensing,” Lim said. “If we are able to capture just a part of our bodies, that is a lot of information to infer to reconstruct the full body.”
Maintaining privacy for bystanders near someone wearing such a sensing device, according to Zhang and Lim, is a legitimate concern when developing these technologies. They claim that BodyTrak reduces privacy concerns for bystanders because the camera is pointed at the user’s body and only collects partial body images of the user. They also acknowledge that today’s smartwatches lack small or powerful enough cameras and sufficient battery life to integrate full body sensing, but that this may change in the future.
