gesturecore bench

gesturecore bench

A workbench for gesturecore, a small library that turns camera hand tracking into clean gesture events: pinches, poses, movements. The bench shows everything the library measures and decides, lets you tune every constant live, and lets you design new gestures without writing code.

1. Quick start

1. Press Start camera (top right) and allow camera access.
2. Raise only your right hand. Its label on the video should say Right. If it says Left, tick swap L/R in the header.
3. Hold an open palm toward the camera. A ring fills around your palm; when it turns green the hand is engaged.
4. Touch your thumb tip to a fingertip and hold. A small ring fills between them; when full you get pinch:start with the finger's name in the Events panel.
5. Engaged and not pinching, make a quick sideways swipe: motion swipeRight appears in the log.
6. Open Gestures to record a pose from your own hand, or build your own movement.

2. How it works

Every camera frame goes through the same one-way pipeline:

The outlined boxes are gesturecore itself. It has no idea there is a camera, a browser or a screen: it receives 21 points per hand and a time, and returns a list of events. That is what lets the same library drive a browser tool, a musical instrument, or run in tests with no camera at all. Everything else on this page (camera, drawing, panels, storage) is the bench, the first program using the library.

Gestures are geometry, not machine learning. MediaPipe does the hard part of finding the hand; gesturecore only measures angles and distances and applies thresholds and timers. Every decision is therefore explainable: the hand panels show the exact numbers each rule looks at.

Two ideas used everywhere

• Hysteresis: an on/off decision uses two thresholds, one to switch on and a looser one to switch off. A value hovering near a single line would otherwise flicker on-off-on every frame.
• Dwell: a state must hold continuously for some time (default 300 ms) before its event fires. Any frame that breaks it resets the timer. The filling rings on the video are these timers.

3. Colour key

Colour carries meaning here; it is never decoration. The same six colours mean the same thing on the video, in the bars, in the log and in these docs. Anything structural (frames, tabs, labels) stays neutral steel.

Left and right hands are told apart by their labels, never by colour, so colour can stay meaningful.

Glow and backdrop

Two colours belong to the interface rather than to the data, so they can never be confused with a measurement:

• Cyan glow = you can click this. Buttons, links, fields, checkboxes and the tab close buttons light up quickly under the mouse.
• Violet glow = a surface, not an action. Tabs, panels, tables and code blocks answer with a slower, softer glow that fades in and out.

The field is a set of slow waves that drift on their own and lean toward what your hands are doing:

Every influence is smoothed heavily, so it reads as atmosphere, never as a readout — for numbers, use the hand panels. It repaints about sixteen times a second on a small canvas, stops when the tab is in the background, and stays still for anyone whose system asks for reduced motion.

You choose where it lives, in Source → Appearance: as the page background, as a panel of its own (Field, where it is drawn sharper and can be watched properly), as both, or as neither. Panels keep a dark scrim over it either way, so text stays readable whatever the waves do.

4. Hand measurements

MediaPipe's depth value z is noisy, so it is used only inside joint angles, never as a distance. Features are computed after smoothing.

5. Engagement

Engagement is a safety catch. A hand in view is not trusted to do anything until it shows the engage pose (default: open palm, held 500 ms). Until then:

• pinch and movement events never fire;
• pose events still fire, since they only describe the hand's shape.

Once engaged, the hand stays engaged whatever it does, until it leaves the camera view (see Losing a hand). This is what stops typing, scratching your face or reaching for a mug from firing gestures.

On screen the outer ring fills amber while you hold the engage pose and turns green when engaged; the skeleton turns green too. Change the pose, or switch the catch off, in Tuning → engage pose.

6. Pinch

For each enabled finger the library measures the thumb-tip distance in spans. With the default settings:

If the thumb opens again before the 300 ms are up, nothing fires and the timer resets. Enable or disable fingers in Tuning → pinch fingers.

7. Poses (hand shapes)

A pose is a list of allowed curl ranges per finger. Fingers not listed are ignored.

{ "name": "fist", "fingers": {
    "index":  { "curl": [0.75, 1] }, "middle": { "curl": [0.75, 1] },
    "ring":   { "curl": [0.75, 1] }, "pinky":  { "curl": [0.75, 1] } } }

• Score: each listed finger scores 1 if its curl is inside the range, fading to 0 when it is pose falloff (0.15) outside. The pose score is the average, shown in the ≈ rows of the hand panel.
• Held pose: the highest-scoring pose that reaches its minScore (default 0.8). Its name goes amber. Equal scores go to the more specific pose — the one constraining more fingers, then the one with narrower ranges — so a thumbs-up beats a fist, which says nothing about the thumb.
• Event: when the held pose stays the same for dwellMs (300 ms), one pose event fires and the name turns green. It fires again only after the pose changes.

Built in: fist, openPalm, point. Make your own in Gestures → Record a pose, which reads your real finger curls over 15 frames and saves ranges of curl ± tolerance. A larger tolerance is more forgiving but may overlap other poses; include the thumb only when it matters, such as a thumbs-up.

8. Movements

A movement describes how the palm centre travels. It fires once when satisfied, and only while engaged and not pinching, so a pinch-drag never doubles as a swipe.

{ "name": "swipeUp", "axis": "y", "direction": -1, "distance": 1.5, "withinMs": 400 }
{ "name": "wave", "axis": "x", "reversals": 3, "distance": 0.5, "withinMs": 1500, "pose": "openPalm" }
{ "name": "push", "axis": "depth", "direction": 1, "distance": 0.3, "withinMs": 500 }

Built in: swipeLeft, swipeRight (1.5 hand-lengths within 400 ms) and wave. The ↝ rows in the hand panel show each movement's progress from 0 to 1 in amber; watch them while you move to see how close you are, then adjust distance or time.

9. Losing a hand

If a hand disappears for more than 150 ms (lostAfterMs), the library closes everything in order: pinch:end if pinching, disengage if engaged, then lost, and forgets that hand. A shorter gap — a finger passing in front, a blurry frame — changes nothing, so a drag survives brief occlusion. Raise the value if tracking drops out often; lower it if a hand that has left the frame feels sticky.

10. The workspace

Every panel is a dockable window (built with dockview):

• Drag a tab onto another panel's edge to split, or onto its tab strip to stack. Drag it out to float it.
• Drag the divider between panels to resize. Double-click a tab to maximize its group.
• Close a panel with the × on its tab; reopen it from the Panels menu in the header, which also has Reset layout.
• The layout is remembered in this browser.

Header

Keys, ignored while typing in a field: Space pause or resume detection · R reset tracking · F mark a false activation during a reliability session.

11. The panels

Camera

• Grey skeleton: the raw points from MediaPipe. Cyan skeleton: the same points after smoothing, turning green once the hand is engaged. The gap between grey and cyan is the smoothing lag; the grey jitter is what smoothing removes.
• Outer ring: engage timer. Inner ring: pose timer. Small ring between thumb and finger: pinch timer. Each fills amber and turns green when it completes.
• The label under the hand reads: hand · pose · engaged · PINCH finger.

Events

Newest first: time in seconds, hand, event. Activations are green, releases white, lost red. Consecutive pinch:move events are folded into the pinch:start line as "moves ×N"; tick each pinch:move to list them individually.

Hand · R and Hand · L

One panel per hand, dimmed when that hand is not visible. Badges show engaged, pinch (with finger) and the held pose — amber while held, green once its event has fired.

Tuning

A control for every library setting; changes apply on the next frame. Reset to defaults, Copy config (to keep or share a tuning) and Import… (paste one back).

Moves

The catalogue: every gesture the core currently knows — the engage gate, each pose, each pinch finger and each movement — with how to perform it and the numbers it is judged by. A row reads now while that gesture is happening, so it doubles as a cheat sheet you can practise against. It lists whatever is in the config, so gestures you add appear here too.

Gestures

Four ways in, and all of them end as the same plain data. A pose or movement whose name already exists is replaced, and Apply on the JSON boxes says exactly what is wrong if anything is.

• Record a pose — hold the shape. The spread of your curls over the hold sets the tolerance, and a finger that wandered is left unconstrained rather than pinned to a meaningless range. Tick include thumb whenever the thumb is what makes the shape different.
• Record a movement — perform it a few times instead of describing it. See below.
• Build a movement by hand — fill in the fields yourself.
• JSON — paste or edit either list directly.

Learning a movement by demonstration

Describing a movement in numbers is guesswork; showing it is not. Give it a name, choose how many takes, and perform the movement. From those takes the bench works out:

• which axis carried it — sideways, up/down, depth or twist — judged in units of a typical movement on each axis, so a big twist is not mistaken for a small drift;
• how many strokes it took, dropping the half-strokes at the start and end of the recording;
• a distance comfortably under your smallest demonstration, and a time comfortably longer than your slowest.

It then replays your takes against the movement it just wrote and loosens it until every take would have fired, so a learnt movement always recognises the demonstrations it came from. The result is shown before anything is added: per-take scores, the axis comparison, and a warning when the axis was not clear-cut — that usually means a bigger, cleaner movement is needed.

Nothing is trained and nothing leaves the machine. The fitted movement is ordinary JSON you can read, tune and share, exactly like one written by hand. The same fitting is part of the library (fitMotion, fitPose), so an app can offer "record your own gesture" without the bench.

Sound

The gesturecore-sound brick. It only listens to what the core produces, so the core behaves identically with it on or off. Browsers keep audio silent until you ask, so nothing sounds until you press Turn sound on.

• Cues — short synthesized sounds for events: a rising tone on engage, a blip per pinch finger (each finger its own note), a click on release, a chime for poses, a swoosh for swipes. Edit them as JSON; on is the event type, and hand, name and finger narrow it.
• Theremin — while your right hand pinches, its height plays notes on a pentatonic scale (hard to make sound wrong), spreading the other fingers makes it louder, and moving sideways changes the tone.
• Controls — the general form of the theremin: any hand measurement steering pitch, volume or brightness while a condition holds.
• Hear the sounds — preview every built-in sound.

Every sound is synthesized on the spot; there are no audio files. The decisions — which sound, which note, how loud — are made by pure code that is tested without audio, and only a thin layer talks to the Web Audio API.

Head

The gesturecore-head brick: a head and face reader. It takes MediaPipe's Face Landmarker points from the same camera frames and turns them into events the same way the core treats hands: measurements, then held states with a dwell and hysteresis, then movements. It is off until you switch it on, because the face model is a download of its own and costs time on every frame.

• yaw, pitch, roll — where the head points, in degrees: turning to your right, looking up, and tilting toward your right shoulder are positive. They come from fitting MediaPipe's canonical face to the forehead, nose, eye corners and sides of the face, so talking or smiling does not move them.
• blink L / R — how closed each of your eyes is, from the eyelid points (0 open, 1 shut). Left and right are your own, whichever way the view is mirrored.
• brows, frown, mouth, smile — from MediaPipe's expression scores (blendshapes), averaged over both sides.
• ≈ expressions — held states: brows raised, frown, mouth open, smile, blink, wink left and right, turn, look up or down, tilt. Each fills amber while it holds and turns green when it starts; the log shows expression:start and expression:end with how long it lasted.
• ↝ movements — nod (down and back up) and shake (side to side), found by the core's own stroke detector running on the head angles.
• Set neutral — makes your usual pose zero and your resting eyes "open", for a camera that is off to one side or eyes that read half closed.

Everything — the settings, the expressions and the movements — is one JSON object in the panel, and the defaults are first guesses to be tuned against real faces. Like the core, the reader is pure code tested without a camera, on synthetic faces turned by exactly known angles.

Reliability

The definition-of-done measurement: fewer than 2 unintended activations per 10 minutes. Start a session, work normally for ten minutes, and press F whenever a pinch or movement fires that you did not mean. Finished sessions are listed underneath. This is a measurement of the core's defaults, not a feature.

Fixtures (test captures)

A fixture is a hand the automated tests replay, so a change in the maths shows up as a failing test rather than a surprise. It is not a picture and not a gesture definition — for the gestures themselves see Moves.

One snapshot is a weak test. Hand tracking is at its best square-on to the camera and degrades on angled and distant views, which is exactly where a gesture quietly stops being recognised. So a fixture holds several views of the same shape — front, turned left and right, tilted up and down, near and far — and the tests require every view to read as the same pose. Views you skip are simply not tested; each one you add tightens the net.

Press Capture every view and follow the prompts, or capture single views with their own buttons. Each view averages several frames. A view that disagrees with the others is flagged in amber before you save it — that disagreement is the finding, and usually means the pose's curl ranges are too tight rather than that the capture is bad. Save to test/fixtures writes the file; then run npm test.

The tests also check the same thing synthetically: each fixture is moved, resized, rotated and mirrored, and must still read as the same pose. That covers what maths can cover; real captures cover what it cannot, namely the tracker's own behaviour at awkward angles.

Source

Camera device and resolution (lower resolution can raise fps), GPU/CPU, and MediaPipe's own confidence thresholds: detection for finding a new hand, presence for deciding a tracked hand is still there, tracking for trusting frame-to-frame tracking. Raising them reduces phantom hands but drops real ones more often. The overlay toggles hide the raw skeleton, the smoothed skeleton or the image.

12. Tuning guide

Change one thing at a time and watch the numbers. When a tuning feels right, Copy config and keep it: those values are what should become the library defaults.

13. Using it in code

import { GestureCore } from 'gesturecore';

const core = new GestureCore({ aspect: 640 / 480, engage: { pose: 'openPalm', dwellMs: 500 } });

// every frame: hands from MediaPipe, time in milliseconds
for (const e of core.update(hands, performance.now())) {
  if (e.type === 'pinch:start') console.log(e.hand, 'pinched', e.finger);
  if (e.type === 'motion' && e.name === 'swipeLeft') history.back();
}

core.getFeatures('Right');   // latest measurements, or null
core.getHandState('Right');  // engaged, timers, pose scores, movement progress
core.setConfig({ dwellMs: 200 });   // live change, same as the sliders

Every event carries type, hand ('Left' | 'Right') and t. Within one frame, hands come Left then Right. Every pinch:start gets exactly one pinch:end, and every engage one disengage. The config is plain JSON — exactly what Copy config gives you. More in the project README.

14. Troubleshooting

15. Files and saving

• Saved in this browser (localStorage): the config (settings, poses, movements), bench preferences, the panel layout and session history. Another browser starts from defaults; clearing site data resets everything.
• Written to the project: only fixture captures, in test/fixtures/.
• Project layout: src/ the library (no browser APIs allowed), test/ the automated tests, bench/ this page, README.md developer notes.
• Commands: npm run bench starts this page, npm test runs the tests, npm run fetch-model downloads the hand model.

↑ back to top