Adithya N Murthy

No. 09of 22LandmarkResearch

Real-time, local-first AI coaching engine

ARI Sync

A coaching engine that turns an exercise video into a live session: it watches you through a webcam, compares your form with the reference, and speaks a correction while you are still moving. Every model runs on one local machine.

When
2026
Status
Built
Team
Registered at RV University as a four-person project with a faculty guide.
Role
Research assistant. My part was the real-time feedback pipeline: pose tracking, form checks, rep counting and the voice coach.

Progress

Progress

Fitness coaching works end to end on one machine. The cooking and building-block versions are scaffolds only, and no controlled benchmark has been run.

What comes next

  1. A controlled benchmark against labeled recordings
  2. Calibration for each user, and a physiotherapist’s review of the angle thresholds

9

joint angles checked on every pose sample

Project overview and paper draft, April 2026Verified. Backed by a file or public page that has been checked.Project overview and paper draft, April 2026

~9.3GB

of models on disk: two 7B language models, Whisper and MediaPipe

Disk footprint. The project's own table puts loaded memory at about 10 GB when both language models are in use.

Project slides and paper draft, April 2026Verified. Backed by a file or public page that has been checked.Project slides and paper draft, April 2026

~2.5ms

per form check

Design target from development instrumentation, not a controlled benchmark.

Project report, April 2026Verified. Backed by a file or public page that has been checked.Project report, April 2026 — The report itself says its latency figures are design targets.

~1.9s

from the start of an error to the first spoken word

Design target from development instrumentation, not a controlled benchmark. Most of it is the language model writing the line.

Project report, April 2026Verified. Backed by a file or public page that has been checked.Project report, April 2026 — The report itself says its latency figures are design targets.

Exhibit 01 · Live demo

Try the pose tracker and the angle checks on your own device

This demo shows a form checker for squats. By default a simulated stick figure, generated by code, performs four squats in a loop. For every sample the checker measures nine joint angles and a four-state machine counts repetitions. Three squats reach full depth and are counted with a form score of 100. One is deliberately shallow: it is not counted, its form score is 60, and the correction reads "Go a little deeper on the way down." A button lets you run the same checker on your own camera; the video is processed on this device only.

Simulated skeletongenerated by code · not a person, not a recording
Exercise: squat
Last rep

Waiting for the first rep.

The verdict is read at the deepest sample of each rep.

Reps counted0
Rep state machine
  1. Waiting
  2. Descending
  3. Bottom
  4. Ascending

Nexta drop of more than 8° in one sample

Left knee angle · last 9 seconds–
Form rules · checked on every sample
  • Depthsmaller knee angle–
  • Back angleshoulder, hip, knee–
  • Left vs rightlargest side difference–

now deepest sample of the last rep

Nine joint angles
L elbow
–
R elbow
–
L shoulder
–
R shoulder
–
L hip
–
R hip
–
L knee
–
R knee
–
Back
–
Measured on your device, just now
Pose model, per frame
camera off
Form check, per sample
–
Frames per second
camera off
Model backend
camera off

Your video is processed on this device and never leaves it.The first start downloads the pose model and its runtime, about 24 MB.

Exhibit 02 · Walk-through

From a video link to a spoken correction

  1. Step 01

    1. Ingest the reference

    You paste a link to an exercise tutorial or upload a video. The server transcribes it, extracts the trainer’s pose at 15 frames per second, splits the video into exercises and writes a flashcard of cues for each one with a local 7B language model.

  2. Step 02

    2. Track you in the browser

    MediaPipe runs as WebAssembly in your browser, so frames never wait on a network round trip. The layout is one browser client and one host machine.

    Architecture diagram. On the left, the client: a React browser app with webcam and MediaPipe, a Three.js companion pane, and a voice queue with flashcards. It connects through a tunnel to one host machine running FastAPI with Socket.IO, a Celery worker, Ollama with two language models, MediaPipe, faster-whisper, Postgres, Redis and local disk storage.
    System layout, from the project report.

  3. Step 03

    3. Check nine joint angles

    A rule-based checker reads nine joint angles on every pose sample and a four-state machine counts repetitions: idle, descent, bottom, ascent.

  4. Step 04

    4. Decide what is worth saying

    A severity gate decides what gets spoken. A major error that lasts half a second is voiced by the coach. A minor one appears as text only.

Exhibit 03 · Chart

What runs on the one machine: model sizes on disk

  1. llava:7b (vision-language model)4.7 GB

  2. qwen2.5:7b (language model)4.0 GB

  3. Whisper small (speech to text)500 MB

  4. Piper voice60 MB

  5. MediaPipe pose6 MB

  6. MediaPipe segmentation1 MB

Scale: 0 to 4.7 GB

File sizes on disk, from the project report: about 9.3 GB in all. The project’s own table puts loaded memory at about 10 GB when both language models are in use.

01

What it is

You give ARI Sync a reference, such as a YouTube exercise tutorial. It transcribes the video, extracts the trainer's pose, splits the video into exercises and writes a flashcard of cues for each one. Then it watches you through your webcam and compares your joint angles with the reference as you move. Fitness is the first use. Versions for cooking and for building with blocks exist as scaffolds only.

02

The hard part

A correction has to arrive while it is still useful, and it has to be worth listening to. Pose tracking runs in the browser as WebAssembly, so frames never wait on a network round trip. A rule-based checker reads nine joint angles on every pose sample. A severity gate decides what gets spoken: a major error that lasts half a second is voiced, and a minor one appears as text only. Without that gate, the project notes say, people tune the coach out within minutes.

03

What was built

An ingestion pipeline on FastAPI and Celery with PostgreSQL and Redis: download, transcription with faster-whisper, pose extraction with MediaPipe, exercise segmentation, and flashcards written by a local 7B language model. A live session over Socket.IO with a four-state rep counter, the form checker and a queued voice coach. A React front end that shows your skeleton beside a background-removed silhouette of the trainer. Two 7B language models, Whisper and MediaPipe all run on one Mac Studio, with no paid cloud AI service.

04

What was found

The project report gives about 2.5 ms for a form check and about 1.9 s from the start of an error to the first spoken word, most of it spent by the language model writing the line. The same report says these are design targets and typical values from development instrumentation, not a controlled benchmark, and they are presented that way here. The demo on this page runs the pose tracker and the angle checks in your browser and shows what your own device measures.

05

What's next

A controlled benchmark against labeled recordings, which the report lists as future work. Calibration for each user, because one set of angle thresholds does not fit every body. And a review of those thresholds by a physiotherapist before anyone relies on them.

Figure

Bar chart of model sizes on disk: qwen2.5:7b 4.0 GB, llava:7b 4.7 GB, Whisper small 500 MB, MediaPipe pose 6 MB, MediaPipe segmentation 1 MB, Piper voice 60 MB.
Model sizes on disk, from the project report.

One question

The answer is on this page. Get it right and it goes in your passport on the map.

Challenge

How long must a major form error last before ARI Sync says it out loud?

Show the answer

Half a second. Half a second. A major error must persist for 0.5 s before the voice coach speaks; a minor one appears as text only. Source: ARI Sync project overview and paper draft, April 2026.

Stack
  • Python
  • FastAPI
  • Socket.IO
  • Celery
  • PostgreSQL
  • Redis
  • MediaPipe (WebAssembly and Python)
  • Ollama (qwen2.5:7b, llava:7b)
  • faster-whisper
  • React
  • TypeScript
  • Three.js