An engineering breakdown of a browser-native, hardware-accelerated projection mapping suite created with ZIM, WebGL, and generative AI collaboration.
Whiskers is a code-defined projection mapping tool built on the ZIM JavaScript Canvas Framework. Unlike static mapping software, Whiskers allows developers and artists to load videos, images, SVGs, and interactive generative code (particle emitters, sound visualizers, animated shapes, ML5 webcam tracking) directly into projection surfaces.
Surfaces are passed into Perspective() and mapped via ProjectionMap(), providing real-time hardware homography warping, dynamic passthrough cropping, multi-mode Bezier masking, GPU Gaussian feathering, multi-layer composite blending, undo/redo history, presentation mode, and full-stage calibration crosshairs.
Direct HTML5 video element piping to WebGL textures without 2D canvas readback bottlenecks. Simultaneously supports interactive ZIM containers, vector shapes, physics simulations, sound-wave animations, and live particle Emitter objects.
Real-time 4-point projective homography mapping. Features selectable texture resolution scaling: 1.0 for native 1:1 pixel crispness, 0.5 for bilinear filtering with a 75% GPU fragment fill-rate savings, and 2.0+ for Retina supersampling.
Integrated Crop Passthrough mode allowing non-warped content to be framed within the perspective bounds. Features independent real-time content scaling (0.2× – 3.0×) and horizontal/vertical pixel translation offsets (±100px).
Stage-level Bezier mask (Blob) supporting 5 curvature modalities (mirror, straight, free, none/corner), vertex insertion, deletion, multi-point selection, and precision keyboard arrow nudging (1px normal / 10px shift).
3-Pass WebGL pipeline: Canvas rasterization → Horizontal Gaussian Blur → Aspect-corrected Vertical Blur → Smoothstep alpha ramp with Interleaved Gradient Noise (IGN) dithering to eliminate 8-bit quantization banding on dark projector beams.
Retained-mode z-index rank manipulation with instant reordering across parent containers. Supports 8 hardware compositing modes: normal, lighten, darken, screen, multiply, add, color burn, and difference.
Snapping crosshair reticle calculating the true optical perspective center via diagonal line-line intersection (p0→p2 × p1→p3), accurately following non-linear perspective foreshortening live at 60 FPS across dragging and key nudges.
50-step deep undo/redo snapshot stack (Ctrl+Z / Ctrl+Y). Complete session state auto-saved to localStorage per project ID, paired with an integrated JSON Import/Export modal pane for cross-machine deployment.
Single display-tree UI built with nested Tile layouts, steppers, sliders, check boxes, and dynamic status-colored borders (red for locked, yellow for hidden, orange for both), collapsible and draggable across the canvas.
Instant distraction-free Presentation mode (P key / double-click) stripping away handles, crosshairs, and panels for live shows, combined with a smooth-animated stage blackout overlay (Space bar).
A breakdown of how this exact project would have progressed using AI collaboration across other popular frameworks and ecosystems, accounting for helper libraries, training data representation, and architectural friction.
Why it succeeded: ZIM’s high semantic density provided pre-built compound classes (Perspective, Blob, Panel, Tile, Slider) sharing a unified scene graph. The AI spent tokens solving projection mapping logic rather than building vector handles or UI components from scratch.
Perspective with quad homography & handlesBlob with cubic Bezier points & tangent sticksPanel, Slider, Stepper)localToGlobal)GPUMaskEngine) to prevent context sprawl
Analysis: PixiJS is the closest 2D WebGL architecture to ZIM. The community plugin pixi-projection handles 2D quad perspective smoothly, and Pixi's sprite/filter pipeline is fast. However, Pixi lacks an interactive Bezier curve editor (no Blob equivalent), requiring the AI to author custom vector handle rigs, hit testing, and drag mechanics using PIXI.Graphics.
pixi-projection plugin for 2D quad warpingAlphaMaskFilter, Gaussian blur)@pixi/ui) is rudimentary; likely requires HTML DOM overlayAnalysis: Benefit from the largest AI training dataset in existence. State management, undo/redo stacks, JSON serialization, and UI panels (via Tweakpane/MUI) are generated effortlessly. The major bottleneck is the **declarative vs. imperative impedance mismatch**: syncing 60 FPS canvas mouse dragging across React state cycles causes frame drops unless bypassed with raw refs, and syncing 2D Bezier overlays across 3D normalized device coordinates (NDC) requires significant boilerplate.
Analysis: Flutter's Material widget library creates the control panel, steppers, and sliders on Day 1. However, Flutter Web has severe architectural friction with low-level WebGL multi-pass FBO rendering. Chaining multiple custom fragment shaders to blur a dynamic Bezier mask and compositing that over a hardware video stream on the Web encounters frequent browser platform bugs.
Matrix4)CustomPainter
Analysis: Excellent for single-canvas creative coding and simple shader experiments. However, p5.js is an **immediate-mode** engine (everything is redrawn inside draw()). Whiskers is an advanced **retained-mode** application. In immediate mode, there is no display hierarchy, no event bubbling on individual shapes, and no coordinate inheritance. The AI has to write a custom object manager, hit-testing engine, and drag system from scratch.
createShader()p5.gui) is inadequate for multi-surface controlsAnalysis: Complete control over every GPU state and byte with zero dependencies. The AI writes GLSL shaders, matrix math, and FBO buffers easily. However, having no framework means the AI must author an entire retained scene graph, matrix hierarchy, event dispatchers, vector hit-testing, cubic Bezier curve evaluation, and UI widgets from scratch. The codebase rapidly explodes in size, triggering AI context degradation.
| Framework / Stack | Dev Time (AI) | Token Burn | Native Projection | Native Bezier Rig | Integrated Canvas UI | Primary Architectural Bottleneck |
|---|---|---|---|---|---|---|
| ZIM (Actual) | 4 – 5 Days | ~2.7M | Built-in | Built-in | Built-in | Smaller training corpus; solution master system prompt. |
| PixiJS (+ pixi-projection) | 5 – 7 Days |
~3.5M | Plugin | None | Minimal | Lacks interactive Bezier curve editor; UI requires DOM overlay. |
| React + Three.js | 6 – 8 Days |
~4.0M | Custom Shader | None | DOM (Tweakpane) | Declarative vs. 60 FPS direct-manipulation drag impedance. |
| p5.js | 10 – 12 Days | ~5.0M | Custom Shader | None | None | Immediate-mode architecture lacks retained scene tree and event bubbling. |
| Flutter Web | 10 – 14 Days | ~5.5M | Custom Shader | None | Built-in | Multi-pass WebGL FBO shader pipeline is poorly supported on Web. |
| Raw JS (Canvas/WebGL) | 14 – 18 Days | ~8.0M | None | None | None | Must author entire scene graph, math, UI, and event system from scratch. |
Prompting velocity is not determined by how much code an LLM can generate, but by how much functionality the framework solves natively in single compound classes. When an engine bundles perspective warping, interactive Beziers, and UI components into a shared coordinate space, the AI spends its attention budget solving application logic rather than reinventing foundational wheels.
When implementing complex GLSL shaders, isolating the raw multi-pass WebGL pipeline into a standalone GPUMaskEngine class was the turning point. It decoupled GPU framebuffers from the vector display list and prevented token-bloat hallucinations.
Large language models degrade in reasoning precision as conversation history expands past 150,000–250,000 tokens. Establishing modular handovers and starting fresh chats kept the AI in high-attention focus across all 18 development cycles.
Early bugs in the reticle/crosshair engine occurred because event listeners tried to mutate internal selection states (selectSide() / selectCorner()). Switching to 100% read-only geometric observation via localToGlobal(0,0) guaranteed zero handle separation.
Having video textures, generative shapes, vector handles, and glassmorphic UI panels coexist within the same canvas hierarchy eliminated the multi-layer DOM-to-WebGL coordinate synchronization bugs common in other stacks.