Local Article-to-Video Chrome Extension (On-Device AI)

Local Article-to-Video Chrome Extension (macOS, On-Device AI)

1. Overview

This project is a macOS-compatible Chrome extension that converts long-form, text-based content into a locally generated video that can be watched instead of read.

The system can run entirely on-device on Apple Silicon Macs (target: MacBook M1 Max). No external APIs, cloud services, or remote inference are permitted. All processing—including summarization, script generation, image generation, audio synthesis, and video rendering—must be performed locally, even if generation takes several minutes.

The solution consists of:

1. A Chrome Extension that handles user interaction and content extraction
2. A local Python companion service that performs all AI inference and video rendering

1. Goals & Non-Goals

Goals

• Turn long articles into watchable videos
• Preserve the informational content and structure of the original text
• Run fully offline after initial model setup
• Prioritize reliability and correctness over speed

Non-Goals (v1)

• Mobile support
• Cloud processing or syncing
• Advanced video editing UI
• Real-time or streaming generation

1. Target Platform & Constraints

• OS: macOS (Apple Silicon) compatible
• Hardware target: M1 Max
• Browser: Google Chrome
• Execution: Fully local, offline-capable
• External APIs: Not allowed
• Processing time: Several minutes acceptable

1. User Experience

Input Methods

• Convert the currently open webpage
• Paste text into a text input field in the extension

User Flow

1. User opens a long article
2. User clicks the Chrome extension
3. User selects input mode and optional settings
4. User clicks "Generate Video"
5. User sees step-by-step progress
6. User watches or downloads the generated video

1. System Architecture

High-Level Architecture

Chrome Extension (UI + Text Extraction) ↓ Local HTTP API (localhost only) ↓ Python Companion Service ↓ Local AI Models + Video Renderer

1. Chrome Extension Responsibilities

UI

• Popup or side-panel interface
• Options:
• • Convert current page
  • Paste text input
  • Basic output preferences (length, tone, voice)
• Progress display with current step
• Error messages and retry controls

Content Extraction

• Extract clean article text from the active tab
• Remove ads, navigation, comments, and unrelated content
• Normalize whitespace and structure
• Handle very long documents by chunking

Job Control

• Send extracted or pasted text to the local service
• Poll job status
• Display logs and progress
• Retrieve final video output

1. Local Python Companion Service

General Requirements

• Runs locally on macOS
• Exposes a localhost-only API
• Handles long-running jobs reliably
• Continues processing even if extension UI closes

1. AI & Media Pipeline (All Local)

Step 1: Text Preprocessing

• Chunk long text into manageable sections
• Preserve headings and structure where possible

Step 2: Summarization

• Generate an information-dense summary
• Preserve key arguments, facts, and narrative flow
• Use local LLMs only

Step 3: Video Script Generation

• Convert summary into a narrated script
• Script must be:
• • Clear and conversational
  • Divided into scenes/slides
  • Aligned with video pacing
• Output includes structured scene metadata

Step 4: Image Generation

• Generate one image per scene
• Images may be:
• • AI-generated (local diffusion models)
  • Abstract or illustrative
• Images must be stored for reuse/debugging

Step 5: Audio Generation

• Generate voiceover narration locally
• Use on-device TTS only
• Voice clarity and realism is important

Step 6: Video Rendering

• Assemble images, audio, and transitions
• Apply simple pan/zoom (Ken Burns style)
• Render to a standard video format (MP4)
• Video length scales with content size

1. Local API Contract (Example)

All endpoints must bind to 127.0.0.1 only.

Endpoints ┌────────┬───────────────────┬───────────────────────────────────────────────┐ │ Method │ Path │ Description │ ├────────┼───────────────────┼───────────────────────────────────────────────┤ │ POST │ /jobs │ Input: { text, title?, sourceUrl?, settings } │ ├────────┼───────────────────┼───────────────────────────────────────────────┤ │ GET │ /jobs/{id} │ Output: { state, step, percent, logs[] } │ ├────────┼───────────────────┼───────────────────────────────────────────────┤ │ GET │ /jobs/{id}/result │ Output: video file or stream │ ├────────┼───────────────────┼───────────────────────────────────────────────┤ │ POST │ /jobs/{id}/cancel │ Cancel a running job │ └────────┴───────────────────┴───────────────────────────────────────────────┘ Security

• Local-only binding
• Shared secret token generated at install
• No remote access

1. Model & Runtime Expectations

Developers may choose specific models, but must:

• Use local inference only
• Support Apple Silicon acceleration (Metal / MPS)

Preferred (not mandatory):

• LLM: llama.cpp or MLX
• Image generation: Stable Diffusion (local, MPS)
• TTS: Piper / Coqui / macOS say fallback
• Video rendering: FFmpeg

1. Storage & Caching

Store:

• Summaries
• Scripts
• Generated images
• Audio files
• Final videos

Support re-runs without regenerating unchanged steps. Clear cache controls (optional).

1. Error Handling & Observability

• Structured logs per job
• Clear error messages surfaced to UI
• Graceful handling of:
• • Model failures
  • Out-of-memory conditions
  • Partial generation failures

1. Code Quality Requirements

• Clean, modular architecture
• Clear separation of concerns:
• • UI
  • Orchestration
  • AI inference
  • Media rendering
• Well-documented code where non-obvious

1. Testing Requirements

• Unit tests for:
• • Text extraction
  • Script segmentation
  • Job orchestration
• Integration tests with small local models or mocks
• Tests runnable via a single command

1. Deliverables

Private GitHub repository containing:

• Chrome extension code
• Python companion service
• Setup scripts
• README including:
• • Architecture overview
  • Local installation instructions
  • Model setup
  • How to run end-to-end
• Example generated video for validation

1. Acceptance Criteria

• Entire pipeline runs without internet access after setup
• No external API calls are made
• Long articles can be converted into watchable videos
• System remains responsive during multi-minute jobs
• Video accurately reflects source content