FERRETGame engine over Babylon.js
Ferret

Media: audio, video, HTML surfaces, capture, section views

Skill ferret-media · download all skills · SKILL.md

When to use#

Music, effects, ambient sound by zone, a screen in the world that plays a stream or shows a page, a recording of the scene as a video file; a frame-exact clip or picture series; a cutaway.

The model#

AudioService has a master gain and four buses (music, sfx, ambient, voice). AmbientZones maps zones to ambient sounds. VideoScreen plays HLS or MP4 onto a plane (loadHls for HLS); HtmlScreen draws a web page onto a plane as a texture. Every media component implements MediaPlayer. SceneRecorder records the engine's canvas (captureStream + MediaRecorder) to one file: MP4 where the browser writes it, else WebM with its Duration put in; start, pause, resume, stop() -> { blob, mime, durationMs, bytes, audio }. The inspector has record.start, record.stop, record.state. captureFrames moves the world exactly 1/fps per frame (nothing dropped, loops close) and writes WebM (WebCodecs) or PNGs; inspector capture. SectionView (package view) cuts meshes by 1 to 6 planes on the GPU with hatched caps (all/any).

Steps#

  1. Add AudioService to the world; define sounds (SoundDef) and play with PlayOptions on a bus.
  2. Ambient: map zone names to sounds in AmbientZones (needs ZoneService, see ferret-interaction-zones).
  3. Video or page screens: add the component to an entity with a plane mesh.
  4. Browsers block audio until a user gesture: start it after the first input.
  5. Recording: world.addService(new SceneRecorder()), then recorder.start() and await recorder.stop(). The sound joins only when the audio is already unlocked at start() (result.audio).
  6. Clip: captureFrames(engine, scene, world, { seconds, fps, width, height, supersample, beforeFrame }). Cutaway: new SectionView(scene, { targets, planes, mode, frame }), toggle active.

Check afterwards#

Each bus can be muted separately; leaving a zone fades its ambient sound; the video plane shows frames (not black); a recording plays back and shows the scene; a capture has seconds * fps frames; a section shows hatched caps.

What usually goes wrong#

  • Audio never starts: no user gesture yet.
  • HLS on a browser without native support: pass the hls.js loader as the API describes.
  • A recording is empty or has no sound: software rendering and encoding (the simulators, a headless browser) deliver the first data about a second after the start, so record for a few seconds; a locked audio context delivers nothing, so a recording started before the first user action has no sound, and the browser may not play an MP4 with AAC back (check the file from outside). Playwright's WebKit has no captureStream.
  • HTML pages that forbid framing or fetching: HtmlScreen draws a fetched page, not a live frame.
  • Capture: no codec for the size -> format: 'png'; code timed by performance.now() runs at wall-clock speed. Section: no caps on open meshes; shadows come from whole meshes.

Files#