WebGL2 replay renderer

01Measured baseline and reason for the rewrite

02Decisions that should not drift mid-implementation

03Target architecture

                              reusable, immutable
                      ┌────── model / anim / stage assets ──────┐
                      │                    HTTP + browser cache   │
                      │                                           ▼
.slp ── C parser ─────┼── replay-scoped timeline.bin ─ HTTP ► browser scene store
                      │                                           ▲
live Slippi events ───┴── committed/replaced batches ─ WebSocket ┘
                                                                  │
                                                deterministic frame clock
                                                                  │
                                       animation evaluation + camera state
                                                                  │
                                                                  ▼
                                       WebGL2 vertex skinning + rasterization
                                                                  │
                                                                  ▼
                                                               canvas

Stateless replay identity

POST /api/replays                         upload, returns immutable replay id
GET  /api/replays/{id}/manifest          identity, players, stage, schemas
GET  /api/replays/{id}/timeline          completed binary state snapshot
WS   /api/replays/{id}/live              live state only

Never: GET /api/set?f=... followed by requests against shared g_active

The server may internally cache parsed replays by id, but request correctness cannot depend on whichever replay another connection selected most recently.

04Proposed module boundaries

Frontend delivery decision

• Use native browser ES modules rather than a production bundle initially.
• Use JSDoc types plus tsc --checkJs for protocol, math, and renderer checking without a runtime build dependency.
• Use Node's test runner for pure timeline, asset, clock, and animation modules.
• Teach the C server to serve allowlisted static files with correct JavaScript, CSS, and binary MIME types.
• Keep shaders as JavaScript string modules initially so shader loading cannot race or require another fetch.

05Completed replay timeline contract

Add a versioned binary endpoint, conceptually:

GET /api/replays/{id}/timeline
Content-Type: application/vnd.melee.timeline
ETag: "replay-hash:timeline-schema"
Vary: Accept-Encoding

header
  magic[4]           "RPL2"
  timeline_schema    u16
  flags              u16
  start_frame        i32
  end_frame          i32
  stage_id           u16
  slot_count         u8
  section offsets    u32...

players[]             identity, character, costume, display metadata
slot_presence[]       bitset per frame
slot[N].x[]           structure-of-arrays, directly typed and indexed
slot[N].y[]
slot[N].animation_index[]
slot[N].animation_frame[]
slot[N].facing[]
slot[N].percent[]
slot[N].shield[]
slot[N].stocks[]
items[]               sparse records grouped by frame
stage_events[]        sparse FoD / Whispy / Stadium events
camera.x[]            required original-camera samples
camera.y[]
camera.zoom[]

Per-player frame — first schema

Why structure-of-arrays

Fields are stored as contiguous arrays rather than repeated frame objects. This improves HTTP compression, permits direct typed-array views, avoids per-frame allocations, and makes random frame lookup a constant set of indexed reads.

Protocol rules

• Every multibyte field has documented byte order and alignment.
• asset_schema, timeline_schema, and live_protocol are separate version numbers.
• Unknown trailing sections are skipped using offsets/lengths.
• Schema mismatch fails visibly; never reinterpret bytes optimistically.
• The endpoint is immutable for a completed replay and supports ETag caching.
• Compression is negotiated by HTTP via Accept-Encoding; Brotli is not part of the binary contract.
• JSON is retained only for small manifests and human-readable errors.

06WebSocket live-state protocol

WS /api/live?f=current.slp

client → HELLO { supported_schema, resume_sequence }
server → SNAPSHOT { replay header + recent committed range }
server → FRAMES   { sequence, first_frame, count, binary records }
server → REPLACE  { sequence, first_frame, count, binary records }
server → GAME_END { final_frame, reason }
either → PING/PONG
server → ERROR    { code, message }

Correctness requirements

• Sequence numbers are monotonic. A gap triggers snapshot recovery.
• REPLACE overwrites prior state by frame number; latest rollback version wins.
• Messages batch frames. Never emit one WebSocket message per player per frame.
• Reconnect resumes by sequence when possible and falls back to a snapshot.
• The client exposes buffer depth but does not couple render timing to socket timing.
• Backpressure has a bound; stale intermediate updates may be consolidated into a fresh snapshot.

Do not casually hand-roll a production WebSocket stack inside the minimal HTTP server. At implementation time, select a small reviewed C WebSocket library or isolate live transport in a dedicated gateway; keep the binary state contract independent of that choice.

07GPU asset delivery and parsing

The existing cache format is suitable for the first browser implementation. It is explicitly serialized big-endian and already contains bones, vertices, weights, indices, primitive groups, materials, textures, animations, and stage sections.

Immutable routes

/assets/v4/models/fox-0.model
/assets/v4/anims/fox-0.anims
/assets/v4/stages/fd.stage

Cache-Control: public, max-age=31536000, immutable
ETag: content hash

Browser parsing strategy

• Use ArrayBuffer + DataView for big-endian headers and records.
• Copy final arrays once into native typed arrays suitable for gl.bufferData.
• Validate counts before allocation and reject truncated/oversized inputs.
• Upload raw RGBA textures directly first; atlas/compressed texture work is deferred.
• Key the in-memory asset cache by schema, character, and costume.

08WebGL2 render architecture

GPU resources per model

Pass order

1. Backdrop clear/gradient.
2. Opaque stage sections with depth writes.
3. Opaque fighter primitive groups with depth writes.
4. Items and simple effects.
5. Transparent primitive groups, sorted at group granularity.
6. Shields and debug overlays.
7. HTML HUD outside WebGL.

Required GL state discipline

• One renderer-owned state cache; application code never calls raw GL.
• Explicit state setup per pass: program, VAO, depth test/write, blend mode, cull mode.
• Development builds check gl.getError() at pass boundaries.
• Handle webglcontextlost and rebuild all resources after restoration.
• Use device-pixel-ratio-aware canvas sizing with a configurable upper bound.

Capability gate before asset loading

• Require a WebGL2 context and sufficient vertex texture image units.
• Record maximum texture size, combined texture units, uniform limits, and shader precision.
• Compile/link the skeletal shader and validate a floating-point bone texture before downloading large assets.
• Show an actionable compatibility screen on failure; never leave an empty black canvas.

Transparency parity policy

The C reference renderer sorts individual triangles. The initial GPU renderer sorts transparent primitive groups, not every triangle. This difference is accepted for the first vertical slice, recorded in image-diff results, and promoted to per-triangle sorting only where the artifact is material.

Projection mapping to preserve

The first shader must match render_pose_profile, not invent a new view:

• Model Z contributes horizontal screen position.
• Model Y contributes vertical screen position.
• Model X supplies painter/depth ordering.
• Negative replay facing mirrors the profile horizontally.
• Replay x/y is the authoritative fighter root; animation root locomotion is not double-applied.

09Animation evaluation and GPU skinning

CPU pose evaluation — port in this order

1. sample_key: step, linear, and guarded cubic Hermite.
2. sample_joint: base SRT decomposition followed by animated channel overrides.
3. eval_pose: local matrices followed by parent-to-child world matrices.
4. Rigid versus weighted skin matrix preparation.
5. Upload matrices to the fighter's floating-point bone texture.

Vertex shader contract

#version 300 es
in vec3 a_position;
in vec2 a_uv;
in vec4 a_color;
in vec4 a_weights;
in uvec4 a_bones;

uniform sampler2D u_boneMatrices;
uniform mat4 u_viewProjection;
uniform vec2 u_replayRoot;
uniform float u_facing;

// Fetch 3x4 affine matrices, blend by weights, preserve rigid-vertex rule,
// convert model profile axes to replay world axes, then apply camera.

Validation invariants

• Animation frame quirk (frame <= 0 ? 1 : frame + 1) matches C.
• Loop wrapping matches action end_frame.
• Root animation translation is suppressed where replay x/y is authoritative.
• Facing changes only the intended profile axis.
• Missing animation uses an explicit, tested fallback.

10Camera, controls, and exact frame clock

Camera state

{
  mode: "free" | "melee" | "fit" | "follow" | "stage",
  centerX: number,
  centerY: number,
  zoom: number,
  targetPort: number | null,
  smoothing: number
}

Controls

• Pointer drag: pan.
• Wheel/pinch: zoom around cursor.
• Double click: reset to selected mode.
• Number keys: follow player.
• Mode button: free, original Melee, fit players, stage, follow.
• Camera control remains active while paused and while scrubbing.

Clock rules

elapsed = now - playbackStartedAt
frame = anchorFrame + floor(elapsed * 60 / 1000)
frame = clamp(frame, startFrame, endFrame)

• Never repeatedly add 1000/60; derive from the anchor to avoid drift.
• Seeking sets an integer frame and resets the time anchor.
• On 120/144 Hz displays, a state may render more than once; simulation remains 60 Hz.
• If one presentation is missed, jump to the correct current replay frame rather than slowing simulation.

11Implementation phases and hard exit gates

12Test plan

Visual differential workflow

1. Render a fixture with the C reference renderer at a fixed camera.
2. Render the same frame with WebGL2 at identical dimensions and camera.
3. Capture the canvas.
4. Compare per-pixel and structural similarity.
5. Classify intentional differences; never simply widen tolerance globally.

13Performance and bandwidth budgets

14Risk register and mitigations

15Rollout and cutover

1. ?renderer=webgl2 development path; existing renderer remains default.
2. Internal visual/performance fixture suite must pass.
3. Add user-visible renderer toggle and collect client errors.
4. Make WebGL2 default while retaining ?renderer=software.
5. After a stable window, remove frame prefetch, frame decoding worker, and raster playback endpoints.
6. Keep a command-line/reference C renderer for golden-image generation.

16Definition of done

• Completed replay playback is fully local after initial state/assets load.
• Live mode is batched, sequence-safe, reconnectable, and rollback-correct.
• WebGL2 reproduces accepted C reference poses and stage output.
• Playback clock advances exactly 60 replay frames per second without drift.
• Reference-machine acceptance shows zero dropped presentations over the defined soak.
• Camera is locally controllable in every playback state.
• Concurrent users can operate on different replay ids without shared mutable selection state.
• Measured performance and bandwidth meet the budgets above.
• Raster-frame streaming is no longer used by the product path.
• The old playback implementation is removed, not merely disabled forever.