Lives under self-quarantine can be quite dull to the point of grueling, and yet still brimming with opportunities for creative things to happen.
My makeshift setup -- an iPad accessing remote Jupyter server running on the Pi from r/raspberry_pi
A few weeks ago I did a thread on Reddit revealing what my setup looked like when I was working on this pet project, HIIT PI, on the side. As much as we all have been relentlessly striving to develop on portable devices as we used to do on a normal desktop computer, it still seems beyond the horizon to this day. But that’s probably a topic for another time.
HIIT PI is a web app that uses machine learning on Raspberry Pi to help track your HIIT workout progress in real time (~30fps). The backend runs everything locally on the Pi while you interact with the app wherever there is a web browser connecting to the same local network as the Pi does, whether it be a laptop, a tablet, or a smartphone.
Here’s a quick demo video.
First things first: What’s HIIT
For those of you not familiar with what the term stands for, HIIT, or High-intensity interval training, is a form of interval training, a cardio workout alternating between short periods of intense exercise with periods of rest or lower-intensity exercise, with no equipments required.
It had not grabbed much of traction, until studies a decade ago by several exercise physiologists proved it could actually deliver the biggest health improvement for your workout time. In fact, the fast-growing culture of people consuming streaming contents on YouTube or TicTok allows people like Pamela Reif from Germany to build an entire business empire out of that in her early 20s. How was I doing when I was at her age? Well, banging my head against a stack of grad textbooks cramming for college exams all the while trying to figure out how to install the latest version of Python on a freshly minted machine. 
There are only two hard things in computer science: removing Python 2 as the default Python on your computer and installing Python 3.
— Vicki Boykis (@vboykis) September 21, 2018
Joking aside, I’m not saying that everybody should do HIIT, or any other kinds of workout as professionally as she does, not to mention that I’m not your professional fitness advisor by any means, but I do firmly hold that a daily routine for some moderate exercises does more good to your wellbeings than being sedentary for an extended period of time hunching over a screen without even attempting to stretch your limbs.
The fail case for WFH is not inefficiency; it's burnout.
— Zachary Perret (@zachperret) April 2, 2020
This is typically true for the sake of your own sanity and productivity if you have no choice but to hunker down at home, due to the current Coronavirus outbreak raging across the globe.
Why HIIT PI
This project was born out of my own necessity, to be completely honest with you, when I started out with HIIT only a couple of months ago. I was like, seriously? No dumbbells, no treadmill, no stationary bike, no worries about local gyms or fitness centers shuting down indefinitely, but nonetheless comparable health benefits? Okay great, I’m jumping right in! And let’s kick it off with an exercise pad and some online tutorials.
After a bit of searching, I put together a playlist of videos that seemed beginner-friendly enough, casted the screen on a wall, and followed them through. I carved out 20 minutes after work in the evening every other day exclusive for HIIT activities.
Having managed to pull it off to the end of videos effortlessly each time, I thought it was kind of lame. It turned out that I was doing it the wrong way the whole time, winding up being short of the required training intensity. Certain movements oftentimes did not finish well to the full extent, or got straight skipped in the middle without me even realizing it.
Doing a series of moves the very first time is quite a different thing from doing them the fiftieth repetition when you struggle to catch your breath. Keeping it at a consistent pace while not losing your balance is way harder than I initially imagined it might be. And that was exactly when this project idea came into being. Besides, why not make doing workouts more of a fun than a toil anyway?
How it works
HIIT PI does no more than two things. It acts as an electronic referee using computer vision to automatically captures and tracks detected poses and movements, and then scores them according to a set of rules and standards for each different type of workouts.
If we think of each workout as a series of consecutive poses, we can easily translate those poses into different states that a computer can understand. 
To trigger a certain state, it measures the geometric properties and spatial relations of various body parts against the rules of that particular state.
Burpees, for example, can be decomposed into three distinct poses/states. You complete these actions in the exact same sequence in order to register as finishing one repitition, or a rep in short.
Under the hood
Video stream / Computer vision
The Pi Camera Module takes care of streaming raw image data from the sensor roughly at 30fps. What we need next is a custom recording stream handler subclassed from PiRGBAnalysis, which contains an analyze method where our pose estimation model (PoseNet in TensorFlow Lite) takes in a video frame and spits out pose confidence scores, keypoint position coordinates, and keypoint confidence scores.
class StreamOutput(picamera.array.PiRGBAnalysis):
def __init__(self, camera, model):
"""Custom streaming output for the PiCamera"""
super().__init__(camera)
self.model = model
self.array = None
self.pose = None
self.inference_time = None
def analyze(self, array):
"""While recording is in progress, analyzes incoming array data"""
self.array = array
self.pose, self.inference_time = self.model.DetectPosesInImage(self.array)
...
Admittedly, certain exercise poses are fairly challenging for the current model to be used out of the box. Cases such as when you lay your body down too close to the ground, or when you’re doing bent knee hamstring stretches at an oblique angle to the camera. Finetuning the model with a custom built dataset tailed for this specific task will definitely alleviate the problems. But for now, due to these limitations and time constraints, there’re only several workouts included in the app to play with, like Push Ups, Toe Taps and Jumping Jacks. Others that might also work will be tested and added later for sure.
Edge computing
There has been evidently an ongoing trend for doing inference on the frontends or edge devices instead of sending sensitive personal data over to the cloud. Take TensorFlow.js, which works in the browser, no internet connection required whatsoever. Also, it enables TensorFlow to utilize WebGL for the browser, achieving up to 100x speedups than vanilla CPUs.
By comparison, TensorFlow Lite serves your model natively on smartphones and edge devices. It’s fast and power efficient for running deep learning models, especially convolutional neural networks, on mobile or even microcontrollers. Concretely, it employs quantization technique on the parameters of the original TensorFlow models, converting 32-bit floats to more efficient 8-bit integers, which means reduced file size and inferencing latency, without sacrificing noticeable accuracy losses on end results.
To push the limit even further, an Edge TPU, designed specifically for matrix multiplications, will be delegated to do almost all the heavy lifting of inferencing, gaining another 10~300x speedups depending on model architectures and your operating environments. Here’s an official documentation about performance benchmarks if you’re really into that. The model inference time tested in this project normally stays around 20~30ms (that’s 30~50fps), which is almost in real-time, if you don’t take into account the video rendering load on CPUs.
Dash / Plotly / Flask
As you might tell from the above, HIIT PI uses Dash, a laudable open source tool from the Plotly team, which is in turn built on top of Flask and React for backend and frontent respectively. It essentially allows you to spawn out full-blown interactive web apps with fancy Plotly charts all in Python or R, without touching any HTML and JavaScript stuff.
Apart from that, you can customize common Flask functionalities however you want, like views and routing, redirects, blueprints, database models, alongside all those awesome plug-and-play extensions available out there for Flask. In fact, the video rendering in HIIT PI requests the custom videiostream endpoint from its Flask server like this.
@server.route("/videostream/<workout>", methods=["GET"])
def videiostream(workout):
user_name = session.get("user_name")
session["workout"] = workout
logger.info(f"Current player: {user_name}")
logger.info(f"Current workout: {workout}")
return Response(
gen(camera, workout), mimetype="multipart/x-mixed-replace; boundary=frame"
)
Even beyond that, Dash has a whole repertoire of features and components that have genuinely expedited developing processes. For example this app uses live-updating callbacks to stream data from the backend directly to Plotly graphs and the indicator widgets next to the video frame window, at an interval of 50ms. The thing worth noting though is that we only extend the data rather than update the entire figure in the callback. Otherwise you won’t be able to refresh with a latency on par with inference.
@app.callback(
[
Output("live-update-graph", "extendData"),
Output("indicator-reps", "children"),
Output("indicator-pace", "children"),
],
[Input("live-update-interval", "n_intervals")],
)
def update_graph(n_intervals):
inference_time = redis_client.lpop("inference_time")
pose_score = redis_client.lpop("pose_score")
data = [{"y": [[inference_time], [pose_score]]}, [0, 1], 200]
reps = redis_client.get("reps")
pace = redis_client.get("pace")
return data, f"{reps:.0f}", f"{pace*30:.1f}" if pace > 0 else "/"
Redis / SQLite
As for the persistant database, the built-in lightweight SQLite coupled with SQLAlchemy fits in snugly well. Which means I won’t bother to configure another Postgres server from scratch. Simpler is better.
On the other hand, Redis seems a solid option for in-memory global data storage and transmission. Data comunicating between multiple Dash conponents, Flask sessions and caches all live in separate databases of the local Redis server on the Pi. And it’ll persist Redis data to disk once the user ends a training session.
The Perfect Time is Now
Just imagining a workout is never the same as actually doing it. With everything put into place, let’s slip on sweatpants, and get it rolling!
Now that we’ve all been adapting to this new reality of social distancing, and living with the uncertainty of not knowing how long this is going to last, start digging into what really matters, something you’re so jazzed up about that it keeps you awake at night. At the end of the day, you might find, in retrospect, the experience of shelter-in-place valued, that you make so much progress on culinary skills your home-cooked sashimi does not taste that bad after all.
UPDATE:
As the project evolves over time through a series of incremental iterations, part of the post may differ from the current implementations. Please check out the open source Github repository for the latest updates.