XHS Video Summarizer
An Agent Skill that turns a Xiaohongshu video post into one structured Markdown summary — caption, tags, engagement, speech, on-screen text and comments — with every model running on the local machine.
- Year
- 2026
- My role
- Product definition · UX · Development · Launch
- Platform
- Skill / Workflow
01
The problem
A short-video post is not readable. The caption, the tags, the counts and the comments are scattered around the page, and the actual content is spoken rather than written. Comparing ten posts on one topic means watching ten videos and taking notes by hand: nothing in the format is searchable, and nothing can be diffed afterwards.
02
Approach
Capture first, summarise from files. A Playwright session keeps the login and pulls the caption, tags, counts, comments and the video itself; whisper.cpp transcribes the speech on CPU; a local vision model reads sampled frames and OCRs the on-screen text; a local text model writes the nine-section summary. Every stage leaves a file behind, and nothing leaves the machine.
03
Key product decisions
01
Files first, model second
Each stage writes its own artefact — metadata.json, transcript.txt, visual_analysis.txt — and the summary is built from those. When a step fails you re-run that step rather than the whole post, which keeps the expensive parts cheap.
02
Degrade, never abort
If the local LLM is unavailable the summary falls back to rule-based stitching, and a missing visual pass does not block the transcript. A half-finished post is still worth more than a failed batch.
03
Nine fixed sections
The output shape is the contract: the same nine headings for every post, so two summaries can be diffed and a batch of ten reads like one document instead of ten.
04
From link to summary
01
Capture
Playwright with a persistent profile: engagement counts parsed out of __INITIAL_STATE__, the video URL matched by pattern, the caption and top comments stored alongside it.
02
Understand
whisper.cpp transcribes the audio on CPU; ffmpeg samples a frame every eight seconds and a local VLM describes and OCRs each batch of frames.
03
Summarise
One summary.md per post, written from the files on disk. Batch mode walks urls.txt and reports what ran, what was skipped and what failed.
05
What I learned
The models were the easy part; the plumbing was the work. Login state, ffmpeg on PATH, model weights that may or may not be running, and a schema stable enough to diff — get those right and the model becomes swappable. Keeping every intermediate result on disk is what made the pipeline debuggable at all.