Programmatic Video Generation
Remotion, Hyperframes, OpenMontage, FFmpeg pipelines
Live Demonstration
A quick look at the system in action — input, processing, output.
Programmatic Video Generation — Short Demo
Problem & Motivation
Why this system needed to exist.
Traditional video editing is manual, repetitive, and doesn't scale. Creating 50 videos with the same template but different data requires 50 manual edits. Programmatic video generation treats video as code: data in, composition out.
I explored: Can we build a system where video compositions are defined in code, version-controlled, and rendered automatically from data sources?
What I Built
The system architecture and core capabilities.
A programmatic video pipeline using Remotion, Hyperframes, OpenMontage, and FFmpeg — each suited to different use cases:
Tools & Their Roles
Remotion
React-based. Complex animations, 3D (R3F), interactivity, parametric compositions. Best for: data visualisations, animated explainers, dynamic overlays.
Hyperframes
HTML/CSS-based. Fast iteration, agent-generatable, lightweight renders. Best for: changelog videos, talking-head overlays, captioned content, social clips.
OpenMontage
Auto-editing from transcripts. Cuts silence, adds b-roll, captions, transitions. Best for: podcast recuts, interview packaging, raw footage → polished.
FFmpeg
The universal glue. Encoding, concatenation, filters, loudness, watermarks, chapters, streaming prep. Every output passes through FFmpeg.
System Architecture
How the components connect and data flows.
┌─────────────────────────────────────────────────────────────────┐
│ DATA SOURCES │
│ JSON/CSV/API → Spreadsheets → Databases → Webhooks │
└────────────────────────────┬──────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────┐
│ COMPOSITION LAYER │
│ ┌──────────┐ ┌────────────┐ ┌──────────────┐ ┌────────────┐ │
│ │ Remotion │ │ Hyperframes│ │ OpenMontage │ │ Custom │ │
│ │ Comp │ │ Comp │ │ Timeline │ │ FFmpeg │ │
│ └────┬─────┘ └─────┬──────┘ └──────┬───────┘ └─────┬──────┘ │
│ │ │ │ │ │
└───────┼──────────────┼───────────────┼───────────────┼────────┘
▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ RENDER FARM │
│ Local GPU (RTX 5070 Ti) → Cloud burst (if needed) │
│ Parallel: Remotion (Node) | Hyperframes (Bun) | FFmpeg (CLI) │
└────────────────────────────┬──────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────┐
│ POST-PROCESS (FFmpeg) │
│ H.264 High Profile • AAC 192k • -14 LUFS • Chapters │
│ Watermark • Thumbnails • Multiple resolutions │
└────────────────────────────┬──────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────┐
│ DELIVERY │
│ YouTube API • S3/CDN • Local Archive • Social APIs │
└─────────────────────────────────────────────────────────────────┘End-to-End Workflow
Step-by-step from input to output.
Example: Weekly Changelog Video
- Input:
changelog.md(structured: Added/Changed/Fixed/Removed) - Parser: Extracts sections, categories, PR links, contributors
- Composer (Hyperframes): Generates composition JSON from template
- Assets: Auto-fetches contributor avatars, PR screenshots, feature images
- Render: Hyperframes CLI → MP4 (2-3 min typical)
- Post: FFmpeg adds chapters, watermark, normalises audio
- Publish: YouTube API upload with generated metadata
Example: Data Visualisation Video
- Input: CSV/JSON dataset (e.g., monthly revenue, server metrics)
- Remotion Comp: Animated charts (Recharts → Remotion), counters, transitions
- Parameters: Duration, colour scheme, highlight thresholds
- Render: Remotion CLI with concurrency control
- Output: 4K/60fps ProRes for archive, H.264 for web
Technology Stack
Tools, frameworks, and services used.
Screenshots & Diagrams
Real captures from the running system.
Placeholder — replace with actual renders:
Technical Decisions & Trade-offs
Key choices made during development and why.
Multiple Tools over One
No single tool excels at everything. Remotion for code-driven precision. Hyperframes for speed/agent-gen. OpenMontage for transcript-based editing. FFmpeg for everything else.
Composition as Code (Version Control)
Video comps live in Git. Diffable, reviewable, reproducible. Template changes propagate to all future renders.
Hyperframes for Agent Generation
HTML/CSS is easier for LLMs to generate than React/Remotion. Hyperframes CLI renders to MP4. Agents can write valid compositions reliably.
FFmpeg as Final Gate
Every render → FFmpeg normalisation. Ensures consistent delivery specs regardless of source tool. Single point of quality control.
Challenges & Solutions
What went wrong and how it was resolved.
Challenge: Remotion Render Time at Scale
Solution: Concurrency limiting (2-3 parallel renders on 12GB VRAM). Cloud burst for batches. Hyperframes for high-volume simple videos.
Challenge: Font Rendering Consistency
Solution: Self-host all fonts (WOFF2). Remotion/Hyperframes/FFmpeg all reference same font files. No system font fallback.
Challenge: Audio Sync in OpenMontage
Solution: WhisperX for word-level alignment. FFmpeg atempo for stretch/compress. Manual offset correction for edge cases.
Challenge: Asset Management at Scale
Solution: Content-addressable storage (hash-based). Deduplication. Manifest files tracking asset→composition dependencies.
Outcome & Lessons Learned
What worked, what didn't, and what I'd do differently.
The pipeline handles: weekly changelogs (auto), YouTube intros/outros (template), data videos (on-demand), podcast recuts (batch). It's not a single tool — it's a toolkit where each job uses the right instrument.
Key Learnings
- HTML/CSS video (Hyperframes) is 10x faster to iterate than React video (Remotion)
- Agents can reliably generate Hyperframes compositions; Remotion needs more guidance
- FFmpeg is the only tool that never disappoints — learn it deeply
- Audio loudness normalisation (-14 LUFS) is non-negotiable for platform compliance
- Chapters/metadata in video file saves hours of manual YouTube Studio work
- Version-controlled compositions = design system for video
Roadmap
- Unified composition schema (JSON) → multiple renderers
- Real-time preview server for composition development
- Automated visual regression testing (pixel diff)
- Template marketplace for common video types
- Live data → live video (streaming compositions)
Note: Active toolkit, not a finished product. Each tool serves a specific niche. Screenshots from development renders.
Technical Walkthrough
Extended video covering architecture, code, and implementation details.
Programmatic Video Generation — Technical Walkthrough
Let's discuss how this approach could work for you
Remote roles, contract work, and collaboration — all on the table.