Relay
Multimodal Video-to-Notes Extraction Pipeline with Evidence-Grounded Citations
01.The Problem & Why Naive Implementations Fail
Generic AI transcription wrappers fail abruptly on short-form videos with no speech or heavy background music. They also hallucinate facts without linking back to verifiable source frames, making outputs unreliable for research and knowledge management.
02.Architectural Solution & Engineering Decisions
Built a dual-producer architecture that combines Whisper audio transcription with scene-cut frame sampling (scored via ffmpeg into contact sheets). An encrypted WebCrypto credential vault provides dynamic fallback across Gemini, Groq, and Ollama when quotas or rate limits hit. Packaged as an Android Trusted Web Activity (TWA) to allow instant ingestion straight from the native mobile share sheet.
Key Architectural Highlights
Dual-Producer Multimodal Processing
Normalizes both audio segments and video scene contact sheets into unified TranscriptSegment records, guaranteeing contextual grounding even on zero-speech visual demos.
// ffmpeg scene detection & Whisper alignment
const sceneFrames = await extractSceneCuts(videoPath, { threshold: 0.35 });
const audioSegments = await whisperTranscribe(audioStream);
const normalizedDoc = alignMultiModalStreams(sceneFrames, audioSegments);Resilient Multi-LLM Failover Vault
WebCrypto AES-256-GCM encrypted credential vault with dynamic fallback chains across providers and accounts, handling HTTP 429 rate limits and 402 quota exhaustion with zero dropped jobs.
Asynchronous Worker Queue
BullMQ and Redis decouple user-facing HTTP ingestion from heavy video egress and LLM inference, streaming live execution progress over WebSockets.