This is a CocoaPod which processes video footage of a user doing an exercise, and provides feedback on the user's form. The general idea is to capture a video and convert it to a timeseries, where each data point contains an array of body point locations in 2D space, along with a confidence level on the accuracy of each body point.
It is already possible to estimate body points (called pose estimation), for example using the project OpenPose. tucan9389 has provided an excellent starting point for the code implemented here, based on his project which predicts body points in real-time.
To translate from pose estimation to technique analysis, we need to train a model on timeseries with correct and incorrect form for various exercises. Richard Yang and Steven Chen from Stanford University have had relative success with Dynamic Time Warping (DTW) to solve this type of problem, as described in this paper.
Additional research by Keogh et al. in their paper "Fast Time Series Classification Using Numerosity Reduction" provides a promising, efficient algorithm to train a kNN model with DTW (with k=1 and a dynamic warping window) for a use case similar to the one described here.
This repo contains a sub-directory named video-classifier with Python code for training an Inflated 3D ConvNet (i3D) on videos of people performing exercises. There is currently no solution for porting a TensorFlow model with 3D architecture into a CoreML model, thus the end-to-end model has not been integrated into the CocoaPod. For more information, see the README.
To run the example project, clone the repo, and run pod install from the Example directory first. You should use Xcode's Legacy Build System to correctly include the videos in Bundle Resources (File → Workspace Settings → Shared Workspace Settings → Build System).
iOS 11, Swift 4.2
TechniqueAnalysis is available through CocoaPods. To install it, simply add the following line to your Podfile:
pod 'TechniqueAnalysis'trevphil, trevor.j.phillips@uconn.edu
TechniqueAnalysis is available under the MIT license. See the LICENSE file for more info.