Local Generative AI Lab
Image, video, music, and multilingual voice generation — all local
Live Demonstration
A quick look at the system in action — input, processing, output.
Local Generative AI Lab — Short Demo
Problem & Motivation
Why this system needed to exist.
Cloud AI APIs are powerful but come with constraints: per-token costs, rate limits, data privacy concerns, latency, and vendor lock-in. For high-volume generative media (images, video, audio), local inference changes the economics entirely.
I built a local-first creative laboratory to explore: What's possible when compute is free (after hardware) and models are fully controllable?
What I Built
The system architecture and core capabilities.
A local generative media stack running on consumer hardware (RTX 5070 Ti 12GB, 32GB RAM):
Capabilities
Image Generation
SDXL, Flux, Pony, custom LoRAs. ComfyUI workflows for consistent characters, style transfer, inpainting, outpainting, ControlNet (depth, pose, canny).
Video Generation
SVD, AnimateDiff, LTX-Video, CogVideoX. Image-to-video, text-to-video, video-to-video. Frame interpolation (RIFE) for smoothness.
Music Generation
AudioCraft/MusicGen, Stable Audio Open, Suno API (hybrid). Full tracks, stems, style conditioning, duration control.
Multilingual Voice
Fish S2.1 Pro, XTTS v2, Piper, StyleTTS2. Voice clones for PT/DE/FR/JA/NL/EN-AU from 30-60s references. Emotional control, prosody.
ComfyUI as the Universal Graph
All media generation flows through ComfyUI — it's the "language" of the lab:
- Images: SDXL/Flux → upscale → face detail → color grade → export
- Video: Image → SVD/AnimateDiff → RIFE interpolation → FFmpeg encode
- Audio: MusicGen → stem separation → FFmpeg mix → loudness normalise
- Voice: Text → TTS → voice conversion → FFmpeg post-process
System Architecture
How the components connect and data flows.
Local-first architecture with optional cloud burst:
┌─────────────────────────────────────────────────────────────┐
│ LOCAL MACHINE (RTX 5070 Ti) │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ ComfyUI │ │ Ollama/ │ │ FFmpeg / │ │
│ │ (Primary) │◀▶│ llama.cpp │◀▶│ Python Scripts │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Model Zoo (Hugging Face Cache) │ │
│ │ SDXL, Flux, SVD, MusicGen, XTTS, Whisper, etc. │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌─────────▼─────────┐
│ Cloud Burst │ (Optional: Replicate, Fal, Together)
│ for heavy lifts │ when local VRAM insufficient
└───────────────────┘Model Management
- Models stored in
~/models/— symlinked to ComfyUI, Ollama, llama.cpp - Quantisation strategy: Q4_K_M for LLMs (speed), FP16 for diffusion (quality)
- Auto-download via Hugging Face Hub on first use
- Version-pinned in
requirements.txt/ ComfyUI custom nodes
End-to-End Workflow
Step-by-step from input to output.
Typical Creative Session
- Ideation: Prompt engineering in ComfyUI UI → iterate on latent space
- Workflow Capture: Save as
.json— becomes reproducible pipeline - Parameterisation: Convert hardcoded values to inputs (prompts, seeds, control images)
- API Exposure: ComfyUI
/promptendpoint → called by Hermes agents - Batch Generation: Agent loops over prompt variations → 50-100 outputs
- Curation: Human selects best → feeds back into LoRA training data
Technology Stack
Tools, frameworks, and services used.
Screenshots & Diagrams
Real captures from the running system.
Placeholder — replace with actual ComfyUI captures:
Technical Decisions & Trade-offs
Key choices made during development and why.
RTX 5070 Ti 12GB VRAM Strategy
Offload strategy: Diffusion models (SDXL/Flux) → FP16 on GPU. LLMs → Q4_K_M split CPU/GPU. Video models → sequential (not parallel) to fit VRAM. CPU offload for overflow.
ComfyUI over Automatic1111
Graph-based = programmable. Agents can generate/modify workflows as JSON. API-first design. Custom nodes ecosystem. A1111 is better for manual use; ComfyUI wins for automation.
Fish S2.1 Pro for Voice
Free, header-based auth, multilingual, emotional range, voice cloning from short ref. Beats local XTTS on prosody. Cloud but acceptable for this use case.
FFmpeg as Universal Glue
Every output passes through FFmpeg: container normalisation, loudness (-14 LUFS for YT), chapter injection, watermark, codec optimisation (H.264 high profile, AAC 192k).
Challenges & Solutions
What went wrong and how it was resolved.
Challenge: VRAM OOM on Flux + SVD Sequential
Solution: Aggressive model offloading (ComfyUI --cpu-offload), sequential queue (not parallel), FP8 quantisation for Flux, batch size 1. Accept slower generation for feasibility.
Challenge: Voice Clone Quality Variance
Solution: Standardised reference recording protocol: 60s, quiet room, consistent mic, read fixed phonetically-balanced text. Fish S2.1 is sensitive to reference quality.
Challenge: Video Temporal Consistency
Solution: AnimateDiff + ControlNet (depth/temporal) for structure. RIFE interpolation (2x/4x) for smoothness. Accept that >4s clips need keyframe conditioning.
Challenge: Model Version Drift
Solution: Pinned model hashes in workflow metadata. ComfyUI workflows embed model identifiers. Weekly huggingface-cli sync script with diff report.
Outcome & Lessons Learned
What worked, what didn't, and what I'd do differently.
The lab produces production-assets for the YouTube channels, client prototypes, and internal tooling. It's a creative sandbox that pays for itself in capability.
Key Learnings
- Local inference economics: $0 marginal cost enables massive iteration (100s of variations)
- ComfyUI workflows are code — version control them, parameterise them, test them
- Voice cloning quality depends 90% on reference audio, 10% on model
- Video generation is still the weak link — SVD/AnimateDiff quality ceiling is visible
- MusicGen is surprisingly good for background tracks; stems enable real mixing
- Multilingual voice from single clone per language works — Fish S2.1 maintains identity
Roadmap
- CogVideoX-5B local (needs 16GB+ VRAM — upgrade path)
- Live2D avatar + lip-sync for talking-head automation
- Automated LoRA training pipeline from curated outputs
- Real-time voice conversion for live streaming
- 3D asset generation (Tripo, Trellis) → Blender → video
Note: This is an active research environment. Results vary by model version, prompt, and hardware. Screenshots are from development sessions.
Technical Walkthrough
Extended video covering architecture, code, and implementation details.
Local Generative AI Lab — Technical Walkthrough
Let's discuss how this approach could work for you
Remote roles, contract work, and collaboration — all on the table.