New · Nov 2026
Lily · Hand-Controlled Point Clouds
I built a browser app that turns a webcam into a controller for 3D sculptures made of light. MediaPipe tracks both hands, and every pinch, swipe, and fist reshapes a particle cloud of up to 220,000 points rendered on the GPU.
Shapes morph inside a custom GLSL vertex shader with four render styles and fingertip repulsion. Hand signals pass through One Euro filters and hysteresis, so gestures feel steady instead of jittery. Everything runs on the device, with no server.
- 220k particles
- Morphed between shapes on the GPU every frame
- 10 hand gestures
- Pinch, two-hand twist, swipe, palm stretch, fist burst, and more, each latched so it fires once
- 7 sculptures
- Generated in code inside a Web Worker, plus import for PLY, glTF, and 3D Gaussian splats
- 39 unit tests
- Run with type checks and the build on every push in GitHub Actions
Stack · TypeScript, Three.js, WebGL (GLSL), MediaPipe, Vite, Vitest