Webshooter

Problem

Writing user documentation or training material for a web app normally means manually clicking through every page and interaction, taking screenshots, and assembling them into a document by hand — a process that goes stale the moment the UI changes. There was no tool that could walk a whole site the way a real user would (including clicking and hovering elements, not just loading pages) and turn that into a ready-to-read documentation gallery without a person capturing every image manually. The problem compounds across a portfolio of sites and internal apps rather than existing once: every one of them eventually needs onboarding material, and re-doing this manual capture process per site, per redesign, is the kind of repeated cost that never shows up as a single line item but adds up across a whole stack of properties.

What was built

Webshooter crawls a target site page by page, and on each page interacts with every meaningful clickable and hoverable element, capturing a before screenshot (with the target element outlined in red) and an after screenshot for each interaction. It compiles the results into an HTML gallery with a filterable sidebar table of contents, full-page and viewport captures per page, before/after pairs grouped by interaction type, and a lightbox for full-size viewing, plus a print-optimized PDF version of the same report. It ships two ways to run it: a one-dependency CLI (crawler.py + reporter.py, Playwright only) for local use, and a FastAPI/HTMX web app that wraps the same engine so a crawl can be started from a browser, watched live, and its output browsed or downloaded — deployed as a standing service on the owner's infrastructure behind its own subdomain and access control.

Technical approach

The crawler and reporter are deliberately kept as separable stages: crawler.py writes a manifest.json index (page metadata, interaction records, bounding boxes) alongside PNGs, and reporter.py reads that manifest independently to emit the HTML/PDF report, so a report can be rebuilt or redesigned from a past crawl without re-crawling the live site. PDF generation reuses Playwright itself rather than a separate PDF library — it points a headless browser at the already-generated HTML report with print-background enabled. Several safety rails are hard constraints in the crawl loop: same-domain-only traversal, a regex-based deny list that skips interactive elements whose labels match destructive actions (logout, delete, remove, unsubscribe), form fields filled with obviously fake sample data that is never submitted, and a fresh page reload before every single interaction so state never drifts or compounds across captures. Authenticated crawling is opt-in per run: a session is captured once locally via a setup command that opens a real browser for a manual login, and the resulting session file is uploaded through the web app's Sessions page rather than credentials ever being typed into the app itself. The web app layer adds a SQLite metadata store for run/session history, a Keychain-backed session store (with an in-memory backend for tests) so captured auth sessions never touch disk unencrypted, and an async job runner with progress reporting and timeout handling wired into the app's lifecycle. It runs as a native launchd service on the host machine rather than in the containerized agent platform specifically so Chromium never has to enter that platform's Docker image, and it's exposed through its own dedicated tunnel isolated from the tunnel carrying other production services, with access gated by identity-based access control plus an app-level shared secret as fallback.

Creative approach

Craft

The report format treats documentation as a reading experience, not a raw file dump: a filterable table-of-contents sidebar, interactions grouped by type (clicks/hovers/forms) rather than dumped chronologically, a full-size lightbox on every image, and CSS specifically tuned for print (sidebar hidden, deliberate page breaks between sections, backgrounds preserved) so the same artifact works as both a browsable web gallery and a printable manual without a second export pass. Every page capture ships two versions rather than one — a full-page scrolled shot and a viewport-only shot — because a training reader needs both "here's the whole page" context and "here's what a real user actually sees on load" without scrolling; picking only one loses one of those two questions. The PDF path was deliberately built on top of the same HTML report instead of a separate PDF library: Playwright points its own headless browser at the already-rendered report with print backgrounds enabled, so the print output is guaranteed to match what a reader sees on screen rather than drifting into a second, separately maintained rendering path.

Reframe

The key idea is capturing 'what happens when you click this' as a first-class artifact, not just 'what does this page look like.' Most site-screenshotting tools stop at full-page captures; this one drives real interactions (hover, click) and pairs a before shot with a red-outlined target element against an after shot, which is what actually makes the output usable as step-by-step training material instead of a static image dump. That same instinct — capture the artifact, not a description of it — extends to how the crawler and reporter are split: the crawler's only output contract is a manifest plus PNGs, and the reporter treats that manifest as its complete input, so redesigning the report format is a reporter-only change that can be tested and iterated against a real past crawl without touching the site being documented again.

Process and what failed

The build moved from a CLI prototype into a properly designed web app with its own spec and TDD plan rather than growing organically, and picked up real infrastructure hardening along the way — a Keychain-backed session store with a swappable in-memory test backend, an isolated deploy tunnel kept separate from the shared production tunnel carrying other live services, and a deliberate decision to run as a native host service specifically to avoid shipping a full Chromium install inside another platform's container image. The safety rails in the crawl loop (same-domain-only traversal, a destructive-action deny list, fake but never-submitted form data, a fresh page reload before every single interaction) exist because the earlier CLI-only version had no such guardrails against an authenticated crawl accidentally triggering a real logout or delete action mid-run; folding the crawler into the web app was the point at which those constraints got written down and enforced rather than relied on as an operator habit. Deployment access itself went through two layers rather than one after the initial build: an identity-gated access policy sits in front as the primary gate, with an app-level shared secret kept as defense in depth that can be disabled once the identity layer is confirmed working, rather than shipping with only one of the two.

Outcome

Webshooter is deployed as a standing internal tool with its own subdomain, launchd service, and dedicated tunnel, callable either as a CLI or through its web app. It functions as one of the owner's internal utility tools rather than a customer-facing product, used to generate documentation and training material for other Evolve-family sites and apps. The deploy runbook that stood it up is itself a six-step, order-dependent sequence — environment setup, secret provisioning into the Keychain, launchd service load with an explicit health-check curl, dedicated tunnel creation and DNS routing, an identity-based access policy, then an end-to-end verification pass that includes running one small public crawl and one authenticated crawl before the tool is considered live — evidence that the "internal tool" framing didn't mean skipping the operational rigor a customer-facing service would get.