Overview

Physical therapy is crucial for rehabilitation following many different types of surgery and injury, but it is often severely hampered by lack of access to therapists and lack of adherence to home therapy regimens. Similarly, wellness training and ergonomics training can be crucial components of preventative medicine, but are often not availed of due to a lack of access to proper expertise and guidance. This project aims to develop a computer vision based mobile system that can help people with accurate physical therapy, fitness training and ergonomics, while letting the medical caregivers track progress and compliance of patients. Our proposed real-time monitoring and guidance system integrates expertise in seemingly disparate disciplines - computer vision, computer gaming, wireless networking, high-dimensional machine learning, and human factors - towards an integrated solution that holds great promise to transform physical therapy, fitness training and ergonomic training through a quantitative process that can be done at home or at the workplace. Fundamental advances in the core disciplines towards successful implementation of the integrated solution include new hand and body pose estimation and tracking algorithms that are robust to interactions between hands, rapid motion, and occlusions, and the development of machine learning and avatar rendering algorithms for sensor fusion and expert-trained guidance logic, for both cloud-based and local usage. The aim is to provide avatar based training and real-time visual guidance on mobile devices and virtual reality (VR) platforms like Oculus and Samsung Gear VR to enable end-users to enhance accuracy, effectiveness, and safety for therapy, fitness and ergonomics applications.

On-Demand Virtual Physical Therapist

 

The above figure shows the architecture and data flow of our proposed on-demand virtual physical therapist (PT) system. The patient can train himself/herself at home where multiple sensors are used to capture the patient’s data and avatars are created to provide visual instructions and feedback. Patient's data are sent to the cloud for performance evaluation and error identification. Based on the patient’s performance and some contextual factors, a machine learning-based task recommendation model is proposed to provide automated task update recommendation for patients. Based on the recommendation result, either a new task or a guidance video will be rendered on the cloud and sent to the patient’s device.  The PT can be remote to supervise the entire process. The proposed virtual PT system has the advantages of providing accurate, on-demand and personalized care. It has the potential of significantly reducing clinic visit requirements while offering continuous care, thereby reducing cost, and expanding care for economically disadvantaged and rural patient populations.

The following demonstration video introduces the prototype of our system.