Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ deckcraft-ui-egui = { path = "crates/ui-egui" }
deckcraft-media = { path = "crates/media" }
# codecs (ported from FilmCraft, same owner and licence)
deckcraft-bitstream = { path = "crates/bitstream" }
deckcraft-aac = { path = "crates/aac" }
deckcraft-h264 = { path = "crates/h264" }
deckcraft-hevc = { path = "crates/hevc" }
deckcraft-vp9 = { path = "crates/vp9" }
Expand Down
19 changes: 19 additions & 0 deletions crates/aac/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[package]
name = "deckcraft-aac"
description = "Clean-room AAC-LC encoder and decoder (ISO/IEC 14496-3), ported from FilmCraft"
version.workspace = true
edition.workspace = true
license.workspace = true
rust-version.workspace = true

[lints]
workspace = true

[features]
default = []
rayon = ["dep:rayon"]

[dependencies]
deckcraft-bitstream = { workspace = true }
thiserror = { workspace = true }
rayon = { workspace = true, optional = true }
191 changes: 191 additions & 0 deletions crates/aac/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
# deckcraft-aac

A clean-room AAC-LC encoder and decoder (ISO/IEC 14496-3, MPEG-4 Audio, Low Complexity profile),
written from the standard and the public psychoacoustics literature. Layer L0: it depends only on
`deckcraft-bitstream` and `thiserror` (plus optional `rayon`), has no `unsafe`, and builds for
`wasm32-unknown-unknown`.

No FFmpeg/FAAC/FDK-AAC/libfaad or other GPL/LGPL code was read or ported. ffmpeg is used only as an
external **test oracle** (decoding our streams, producing reference streams).

## API

```rust
use deckcraft_aac::{Decoder, Encoder, EncoderConfig, BitrateMode, AdtsHeader, split_adts};

// Encoder: planar f32 (±1.0) → raw access units
let mut enc = Encoder::new(EncoderConfig::cbr(48_000, 2, 128_000))?; // or EncoderConfig::vbr(48_000, 2, 3.0)
let mut aus: Vec<Vec<u8>> = enc.encode(&[&left, &right]); // 0..n AUs per call, any chunk size
aus.extend(enc.flush()); // zero-pads and drains the look-ahead
let asc = enc.audio_specific_config(); // bytes for the MP4 `esds`
let delay = enc.priming_samples(); // 1024 → edit list media_time
let stats = enc.stats(); // frames, bytes, bitrate(), short blocks, M/S, TNS

// Decoder: AudioSpecificConfig + raw AUs → planar f32, 1024 samples per channel per AU
let mut dec = Decoder::new(&asc)?;
let pcm: Vec<Vec<f32>> = dec.decode(&aus[0])?;

// ADTS
let frames = split_adts(&adts_bytes)?; // [(AdtsHeader, raw payload)]
let mut dec = Decoder::from_adts(&frames[0].0)?; // channel config 0 → layout from the in-band PCE
```

`EncoderConfig` fields: `sample_rate` (the 13 AAC rates, 7.35–96 kHz), `channels` (1–8), `mode`
(`BitrateMode::Cbr(bps)` / `BitrateMode::Vbr(quality 1.0..=5.0)`), `bandwidth` (Hz, `None` = automatic
from bitrate/quality), `tns`, `ms_stereo`, `window_shape` (sine/KBD), `adts` (prefix each AU with a
7-byte ADTS header).

**Channel order** is AAC element order, for both encoder input and decoder output:

| channels | configuration | order |
|---|---|---|
| 1 | 1 | C |
| 2 | 2 | L R |
| 3 | 3 | C L R |
| 4 | 4 | C L R Cs |
| 5 | 5 | C L R Ls Rs |
| 6 | 6 | C L R Ls Rs LFE |
| 7 | 0 + PCE | C L R Ls Rs Cs LFE (PCE in the ASC; repeated in every AU when ADTS-framed) |
| 8 | 7 | C Lc Rc L R Ls Rs LFE |

The decoder also reads configurations 11 (6.1: C L R Ls Rs Cs LFE) and 12 (7.1: C L R Ls Rs Lsr Rsr
LFE) from ISO/IEC 14496-3 Amd 4; ffmpeg 9 writes 7.1 AAC as configuration 12.

`Decoder::layout()` reports the element layout (it follows a PCE for configuration 0).

## Encoder

Per 1024-sample frame and channel element:

1. **Transient detection**: energy of the first-difference (high-passed) signal in 128-sample
sub-blocks against the mean of the previous eight; one frame of look-ahead selects
`ONLY_LONG` / `LONG_START` / `EIGHT_SHORT` / `LONG_STOP`. Short windows are grouped by energy
similarity (an attack starts a new group). LFE is always long.
2. **MDCT** (2048/256 points) via a DCT-IV on an N/4-point radix-2 complex FFT; sine or KBD
(α = 4 long, 6 short) windows.
3. **Psychoacoustic model** in the MDCT domain: band energies, tonality from the spectral flatness of
a ≥16-line neighbourhood, Schroeder's spreading function in the Bark domain, tonality-dependent
masking offset (30 dB tonal … 6 dB noise), Terhardt threshold in quiet (capped at 40 dB SPL) as a
floor, pre-echo control (threshold ≤ 2× previous frame), perceptual entropy.
4. **TNS** on long windows: Levinson–Durbin (order 8) over the spectrum above 1.4 kHz; applied when the
prediction gain exceeds 1.4, with 4-bit reflection coefficients.
5. **M/S** per band (and group) for channel pairs (always common window): chosen when the estimated
perceptual cost of M and S (with the stricter L/R threshold, halved) is below that of L and R.
6. **Rate–distortion loop**: one multiplier λ scales every band's allowed noise. Each band gets the
coarsest scalefactor whose *measured* quantisation error meets its target (binary search,
`0.4054` rounding), scalefactor differences are kept within ±60, codebooks and sections are chosen
by dynamic programming over all 12 codebook states, and λ is bracketed and bisected until the frame
fits its budget (CBR) or fixed from the quality (VBR).
7. **Bit reservoir** (CBR): frame budgets follow the perceptual entropy (short-window frames get extra),
bounded by the 6144-bit-per-channel decoder buffer; overflow is padded with fill elements and the
final frame drains the reservoir, so the stream is exactly `frames × bitrate × 1024 / rate`.

Bandwidth: automatic from bitrate per full-bandwidth channel (e.g. 64 kbps/ch → 16 kHz, 96 → 19 kHz,
≥144 → 22 kHz) or from quality in VBR.

Delay: 1024 priming samples (`priming_samples()`); after `flush()` the decoded stream minus the priming
covers every input sample (`ceil(n / 1024) + 1` access units).

Performance (Apple Silicon, release, single thread): 10 s of 44.1 kHz stereo encodes at ~85× realtime
and decodes at ~1000× realtime. With the `rayon` feature channels are quantised in parallel inside the
rate loop.

## Decoder

Full AAC-LC syntax: SCE/CPE/LFE, DSE, FIL, PCE (reconfigures a configuration-0 stream), all window
sequences and both window shapes (shape of the previous frame for the left half), section data,
scalefactors, pulse data, TNS (all-pole synthesis, long and short), M/S (per band / all bands),
intensity stereo (in/out of phase, M/S-inverted), PNS (with correlated noise for `ms_used` noise bands
as specified), IMDCT and overlap-add. Output is planar f32 at ±1.0 full scale. All reads are bounds
checked: malformed access units return `Err`, never panic.

## Test results

All produced by `cargo test -p deckcraft-aac -- --nocapture` (ffmpeg 9.0 at `/opt/homebrew/bin` as
oracle; tests skip when it is absent). Fixtures go to `target/fixtures/aac/`.

### Encoder quality / bitrate (2 s synthetic signals, decoded by ffmpeg)

SNR is full-band; bwSNR is measured after low-passing both signals at the coded bandwidth;
segSNR averages 1024-sample segments above −50 dBFS (clamped to −10…60 dB). "us-vs-ff" is the
largest difference between our decoder and ffmpeg's on the same stream. Every stream decodes without
ffmpeg errors, no output clips (peak < 1.0), and CBR is within ±0.1% of target (test limit ±5%).

| signal | kbps | actual | bw Hz | SNR | bwSNR | segSNR | short frames | us-vs-ff |
|---|---|---|---|---|---|---|---|---|
| sweep 2ch 44.1k | 64 | 64.00 | 11000 | 12.1 | 36.1 | 43.6 | 1 | 5e-7 |
| sweep 2ch 44.1k | 128 | 127.92 | 16000 | 26.1 | 27.3 | 54.2 | 1 | 3e-7 |
| sweep 2ch 44.1k | 320 | 319.90 | 21609 | 46.6 | 46.6 | 59.6 | 1 | 3e-7 |
| white noise 2ch 44.1k | 64 | 64.00 | 11000 | 1.3 | 3.3 | 1.3 | 1 | 5e-7 |
| white noise 2ch 44.1k | 320 | 319.91 | 21609 | 16.4 | 16.5 | 16.6 | 1 | 1.5e-6 |
| speech-like 2ch 48k | 64 | 64.00 | 11000 | 13.6 | 14.0 | 15.2 | 13 | 2e-7 |
| speech-like 2ch 48k | 128 | 127.98 | 16000 | 21.3 | 23.0 | 20.7 | 13 | 2e-7 |
| speech-like 2ch 48k | 320 | 319.96 | 22000 | 31.2 | 31.5 | 31.3 | 13 | 2e-7 |
| chords 2ch 44.1k | 64 | 64.01 | 11000 | 12.4 | 12.4 | 20.6 | 4 | 2e-7 |
| chords 2ch 44.1k | 128 | 128.00 | 16000 | 20.1 | 20.1 | 38.5 | 4 | 2e-7 |
| chords 2ch 44.1k | 320 | 319.99 | 21609 | 35.9 | 35.9 | 55.5 | 4 | 2e-7 |
| clicks 2ch 48k | 64 | 64.00 | 11000 | 10.4 | 11.2 | 48.6 | 11 | 9e-8 |
| clicks 2ch 48k | 128 | 128.00 | 16000 | 20.2 | 25.1 | 53.9 | 11 | 2e-7 |
| clicks 2ch 48k | 320 | 319.95 | 22000 | 39.6 | 39.6 | 58.0 | 11 | 1e-7 |
| speech-like 1ch 44.1k | 64 | 64.00 | 16000 | 18.4 | 19.2 | 18.0 | 15 | 2e-7 |
| chords 5.1 48k | 320 | 320.00 | 16000 | 27.2 | 27.2 | 37.9 | 9 | 2e-7 |
| chords 3/4/5/7/8ch 48k | 192–448 | exact | 16000 | 26–28 | 26–28 | 38 | 7–13 | 2e-7 |
| chords 2ch 8 k … 96 k (10 rates) | 24–256 | exact | 3.9–21 k | 13–38 | | 17–47 | | ≤3e-7 |

Reference points from ffmpeg's own AAC encoder on the same signals (stereo 48 kHz, segSNR):
sweep 128k 39.7, speech 128k 21.2, chords 128k 32.8, clicks 128k 43.3 (ours: 52/20/39/54). Pure SNR
is not a perceptual measure: the model deliberately leaves noise under the masking threshold, and
full-band white noise is the worst case by design.

VBR (`Vbr(q)`, 44.1 kHz stereo): chords q1 62 kbps / q3 139 / q5 225 (segSNR 18 / 39 / 51);
speech-like q1 42 / q3 166 / q5 272 kbps (segSNR 13 / 29 / 41). Bitrate and quality grow
monotonically with q.

### Decoder accuracy vs ffmpeg (streams from `ffmpeg -c:a aac`, PNS off)

| stream | max abs error |
|---|---|
| stereo 44.1k 128k / 48k 256k / 96k 256k | 1.2e-7 / 1.2e-7 / 1.3e-7 |
| mono 48k 64k / 22.05k 32k / 16k 24k / 8k 16k | 1.5e-7 / 1.2e-7 / 1.2e-7 / 8.9e-8 |
| stereo 44.1k 96k with intensity stereo + M/S + TNS | 1.2e-7 |
| stereo 32k 48k with intensity stereo | 1.2e-7 |
| 3.0, 5.1 (320k), 7.1 wide with PCE (448k) | 1.2e-7, 1.5e-7, 1.3e-7 |
| stereo 44.1k 160k `-aac_coder fast` | 1.2e-7 |

Our own encoder's streams: ≤1.6e-6 against ffmpeg for every case above. PNS (ffmpeg's encoder does not
emit it, so streams are hand-built): total energy within 0.15 dB of ffmpeg; per-frame energy within
0.8 dB (random vectors differ). For noise bands flagged `ms_used` we produce identical L/R noise as
the standard specifies; ffmpeg produces independent noise there.

### Robustness

12 000 mutated access units (bit flips, byte overwrites, truncation, extension, splices, random
buffers) over mono/stereo/5.1/7-channel streams, 20 000 random `AudioSpecificConfig`s and 2 000 random
ADTS streams: no panics. The encoder accepts NaN/±inf/over-range input, any chunk size, and missing
channels.

## Clean-room notes

- Syntax, tables (scalefactor bands, TNS limits), filterbank, TNS, PNS, M/S and intensity stereo are
implemented from ISO/IEC 14496-3.
- The Huffman codebooks (`src/huffman_tables.rs`) are the normative values of the standard's Annex
4.A, recovered by black-box probing of the ffmpeg *binary* as an oracle: `tools/` contains the
Python (stdlib-only) scripts that craft single-frame streams repeating a candidate bit string, fit the
decoded output to the IMDCT basis, and walk the code tree. Every table is verified to be a complete
prefix code (Kraft sum exactly 1) covering each index once (unit test), and the decoder matches
ffmpeg to ~1e-7 on real streams.
- Psychoacoustics follow textbook models (Bark scale, Schroeder spreading, Terhardt ATH, Johnston-style
tonality/SFM offsets, perceptual entropy).

## Gaps / not implemented

- HE-AAC v1/v2 (SBR/PS): explicit signalling is parsed and only the AAC-LC core is decoded (half-rate,
no SBR); implicit SBR fill elements are skipped.
- AAC Main (prediction), SSR (gain control), LTP, ER/LD/ELD object types, coupling channel elements
(CCE → `Err(Unsupported)`), 960-sample frames.
- Encoder: no PNS or intensity stereo (decoder supports both), no TNS on short windows, no pulse data;
window shape is fixed per stream (sine or KBD). Short blocks at onsets cost segmental SNR on
speech-like material relative to long windows + TNS; perceptually they avoid pre-echo.
- No error concealment: a corrupt AU returns `Err`; the caller may substitute silence.
- ADTS with CRC is parsed (header skipped, CRC not verified); multi-block ADTS frames are split per
frame only.
Loading
Loading