Skip to content

Video: embedded video clips play in MP4/MOV export - #622

Open
NikitkaRa wants to merge 3 commits into
storytold:mainfrom
NikitkaRa:feat/video-clips-export
Open

NikitkaRa wants to merge 3 commits into
storytold:mainfrom
NikitkaRa:feat/video-clips-export

Conversation

@NikitkaRa

Copy link
Copy Markdown
Contributor

Stacked on #616 (which is itself stacked on #608). Review only the last commit. No issue: this covers the "video clips show their poster frame" part of the docs/gaps.md video export entry.

Summary

  • In exported MP4/MOV, an embedded video clip that plays (autoplay, or a Media effect on the show timeline) now shows its moving pictures instead of the poster frame.
  • Decoding uses the repo's existing deckcraft_media::VideoDecoder, the same one the slide show uses (H.264/HEVC/VP9/AV1, pure Rust). No new dependencies.
  • Pictures are decoded in order as the export's clock reaches them, with a seek on loops or jumps back.
  • Trim, loop and rewind settings apply. After the clip ends its last picture stays on screen, unless the clip is set to rewind, which goes back to the poster.
  • A still or held slide is rendered every frame only while one of its clips is playing. Otherwise it is still drawn once.
  • deckcraft_render::RenderOpts gains video_frames: &[(ShapeId, VideoFrame)] (straight RGBA). The renderer stretches that picture over the media shape's frame in place of the poster, so crop, rotation and animation state are all handled by the normal rendering.
  • A clip that can't be decoded keeps its poster and is listed in the export's warnings.
  • Docs: the gaps.md entry is narrowed to animated GIF (Export: animated GIF of the deck (Export… dialog, CLI, MCP) (#171) #177) plus the missing Export dialog page, and the MP4/MOV parity row is updated.

Test plan

  • cargo xtask ci (fmt, clippy, test, assets, layers, wasm)
  • New test video_clips_play_their_pictures: a red-then-blue H.264 clip, built with the in-repo encoder, autoplays. The export is decoded back and checked frame by frame: red for the first second, blue after that, and the last picture held.
  • Checked by eye: an ffmpeg testsrc2 clip autoplays on the sample deck (540p25). The frames step through the pattern, and the clip's own timestamp in the picture matches the export clock: 1.88 s into the clip at 2.6 s of video, after the slide's ~0.7 s transition. The last picture holds after the clip's 4 s.

Parity checklist: docs/file-format-parity.md MP4/MOV row and docs/gaps.md updated.

NikitkaRa and others added 3 commits October 16, 2026 19:48
file.export writes .mp4/.mov: each slide's transition, then the slide
held for its "advance after" time (else secondsPerSlide), at a chosen
height (480p to 4K) and frame rate, hidden slides left out unless asked.

Frames come from the slide and transition renderers and are encoded by
a new Constrained Baseline H.264 encoder in deckcraft-h264 (CAVLC,
Intra 16x16 with V/H/DC prediction, unchanged macroblocks skipped,
deblocking off), checked bit-exact against our decoder, and muxed with
deckcraft-isobmff (faststart). A key frame starts every slide and
transition and at least every 2 s. No external tool is needed.

Co-authored-by: Cursor <cursoragent@cursor.com>
Exported video now plays each slide's in-slide animations on the slide-show
timeline (a secondsPerClick pause before each click step; with/after-previous
and timed effects play as in the show) and carries a sound track: embedded
audio (autoplay and Media effects, trim, fades, volume, loop, play across
slides), narration clips and transition sounds, mixed to 48 kHz stereo.

Sound is AAC-LC by default via deckcraft-aac, a pure-Rust encoder ported from
FilmCraft (layer 1, no new third-party deps), or 16-bit PCM in .mov
(audio: pcm); audio: none drops it. Video clips still show their poster frame.

Co-authored-by: Cursor <cursoragent@cursor.com>
A video clip that plays (autoplay or a Media effect) now shows its pictures
in the exported video instead of its poster frame. Pictures come from
deckcraft-media's VideoDecoder (the slide show's own decoder), decoded in
order as the export's clock reaches them, with trim, loop and rewind honoured;
after the end the last picture stays unless the clip rewinds. A still or held
slide is rendered per frame only while a clip on it plays.

RenderOpts gains video_frames (straight RGBA per media shape), which the
renderer draws over the clip's frame in place of the poster. Clips that can't
be decoded keep the poster and are listed in the export's warnings.

Co-authored-by: Cursor <cursoragent@cursor.com>

@FredApps FredApps left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReviewerA — source-read COMMENT (not an approval)

I read the head of the diff: embedded video clips play in MP4 and MOV export. The lock hunk is the same deckcraft-aac addition as in deckcraft#616, #608 and the others of the series by this author, so this is a later PR of one stack (H.264 encoder, AAC and sound, video clips, Create a Video page, Export dialog entry, letterboxing) and the stack should merge in order or be sent as one PR. Playing an embedded clip in an export means decoding the user's video, which needs a decoder for arbitrary codecs; the PR should say which codecs it decodes and with what (a dependency or the platform). The patent question on encoders (H.264, AAC) is the maintainers' to decide. I read the lock hunk. I have not run it, and no exact-head CI result exists for this head. Nothing was run.

I ran nothing and have no exact-head CI result for this PR, so this is not an approval. Where a diff was long I read its code hunks rather than every generated or documentation line.

Status: waiting_author.

@aiongg aiongg added the missing feature The reference application has it and we do not, or only part of it label Nov 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

missing feature The reference application has it and we do not, or only part of it

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants