Viceroy

Problem

Passing a state licensing exam on the first attempt is mostly a measurement problem before it is a knowledge problem: most self-study tools report a completion percentage against a course, not whether a candidate would actually pass today, and none flag which concepts have been learned and then quietly forgotten. Generic flashcard and quiz apps also have no way to guarantee that the questions they generate are accurate against the specific regulatory text being tested, so a bank produced by an unreviewed generator risks teaching the wrong answer with total confidence. A separate, subtler failure mode common to auto-generated question banks is a structural tell: if a generator defaults its answer key the same way often enough, a test-taker can learn the generator's habit instead of the material, which defeats the purpose of practicing at all. Viceroy was built to solve both problems for one exam candidate: build a large, citation-traceable question bank from a real course text, verify every answer independently before trusting it, and turn raw accuracy into an honest, decaying readiness signal rather than a completion bar.

What was built

Viceroy is a two-stage system with a single SQLite database as the seam between them. An offline content pipeline extracts a licensed course text into 85 line-cited sections and a glossary, groups them into generation units, and produces a bank of 877 active questions (879 generated, a small number retired for quality) alongside a concept graph, number-fact drills, and exam-trap notes, every fact carrying a file-and-line citation back to source. Every generated answer key is blind re-answered by a separate verification pass that sees only the cited passage, not the generator's key, with disagreements flagged rather than silently kept; the verifier itself is mutation-tested by seeding corrupted keys and checking that the blind pass still catches them. A FastAPI and Jinja2 web application, with plain JavaScript for interactive partials, reads the resulting database and serves a dozen surfaces: a readiness dashboard, adaptive practice with confidence capture and mistake classification, a spaced-repetition review queue, a full timed exam simulator with pacing and stamina tracking, a parameterized math lab for loan-math calculations, and a coaching surface grounded in retrieved course text. Three independent audit rounds found and fixed thirty issues before the app was pushed to a passcode-gated deployment.

Technical approach

The content pipeline and the runtime app are deliberately decoupled through one file: the SQLite database. The pipeline (extract, glossary, unit assembly, then a generation and blind-verification pass, then build) treats the course text as the only privileged input and writes everything else, questions, concepts, traps, number facts, comparisons, as derived, citable rows. The rebuild is transaction-scoped for safety: it opens a manual immediate transaction so existing readers keep a consistent snapshot mid-rebuild, then drops and recreates only the content tables. A discovered gotcha shaped the implementation directly: SQLite's executescript() call implicitly commits any open transaction before it runs, which would silently break the atomicity the rebuild depends on, so the combined content-and-runtime schema is instead split on statement boundaries and executed one statement at a time inside the still-open transaction. Runtime tables (attempts, answers, spaced-repetition cards) are preserved across a content rebuild by snapshotting old row IDs against natural keys (unit and question index, concept name, glossary term) before the drop, so a re-run of the generation pipeline does not orphan a user's answer history.

Determinism is enforced at multiple points. Generators initially wrote the correct option at index 0 in 702 of 879 questions, an exploitable tell caught only by the blind verification pass and invisible to a reviewer checking any single question for accuracy. The fix is a per-question option shuffle seeded from a stable key (unit and index) via a seeded random generator, so a rebuild reorders nothing that has already been shown to a user, and repeated regenerations produce a stable, testable permutation. The six scoring engines (mastery, retention, weakness-impact, readiness, exam assembly, error classification) are specified as closed-form formulas rather than a trained model, and are covered by 31 pytest tests that make zero external calls, including the AI-adjacent code paths. The verifier's own mutation gate requires catching at least 80% of seeded corrupted-key probes or the content build fails outright, treating the verification guard itself as a component that needs its own test rather than a check assumed to work. The one runtime AI feature, a grounded coach, shells out to the Claude CLI with retrieved course sections as context and explicitly refuses to answer when retrieval returns nothing, preserving the guarantee that nothing outside the licensed course text is asserted as fact. The FastAPI app enforces its own auth: an HMAC-derived session cookie is checked against an environment-provided passcode on every request outside a short exemption list (login, static assets, health check), and the gate fails closed whenever that passcode is unset.

Creative approach

Craft

The interface deliberately leads with a single readiness number rather than a completion percentage, because completion measures effort and readiness measures the thing the candidate actually needs to know before exam day: would I pass right now. The home surface pairs that gauge with a forgetting radar, surfacing concepts trending toward being forgotten before they lapse rather than after, and a single prescriptive next action instead of an open menu of study modes. The exam simulator mirrors the real test's shape deliberately, matching its full question count, its full time limit, and its scored-versus-unscored split, so pacing and stamina are trained under the same constraints as exam day rather than in an idealized untimed quiz. A parameterized math lab exists as its own surface, separate from the general question bank, because loan-math formulas (loan-to-value, debt-to-income, per-diem interest, adjustable-rate caps) are a distinct failure mode from recall questions and needed drilling that generates fresh numbers rather than the same memorized examples every time.

Reframe

The reframe is treating an AI-generated question bank as a manufacturing-defect problem rather than a content problem: a generator develops consistent, exploitable habits, like always keying the right answer first, that a human reviewer skimming for correctness will not notice, because the habit is about position, not content. The fix is process, not vigilance: a structurally separate blind verification pass that never sees the proposed key, followed by a mutation test on the verifier itself, so the guard's own sharpness is measured rather than assumed. That two-layer check caught the 702-of-879 pattern that a straightforward accuracy review would have missed entirely, since every individual answer was in fact correct; the defect was in the distribution of correct-answer positions across the bank, not in any single question. The same discipline extends to runtime: every generated fact must trace to a citation checkable by line number, and the one AI feature that runs live is built to refuse rather than guess when its retrieval comes up empty.

Process and what failed

The generator's own bias was the first real failure: the question bank read as though it had been carefully hand-written, and only the blind verification step against source text, not a spot-check of the key, surfaced that 702 of 879 questions kept the correct option in the same position, a pattern invisible to anyone reviewing individual questions for accuracy. Content rebuilds were originally at risk of losing all runtime history, logged answers and spaced-repetition scheduling state, every time the question bank was regenerated, since a straightforward drop-and-recreate has no concept of 'the same question' across a rebuild; the fix maps old rows to new ones by natural key (unit, index, concept name, term) rather than by database ID, which is not stable across a rebuild. The transaction approach also went through a wrong turn: using SQLite's executescript() for the combined schema script looked correct until it was discovered that the call commits any open transaction on its own, which would have quietly broken the atomic single-transaction rebuild the design depended on; the fix was to give up the convenience call and execute each statement individually inside the manually managed transaction.

Outcome

Viceroy shipped as a working, deployed personal tool: 877 of 879 generated questions are active after quality retirement, each blind-verified against its cited passage, and the verifier's own seeded mutation probes were all caught in the recorded run. Thirty-one engine tests pass without a single AI call, and three independent audit rounds found and fixed thirty issues before deployment. As of the last recorded update, the runtime tables were still clean: the diagnostic assessment that seeds the readiness engine had not yet been taken, so the system is proven on content accuracy and engine correctness but not yet on a real study cycle running through it. Voice and podcast study modes, a regulatory-update monitor, and state coverage beyond the one licensing jurisdiction it was built for are explicitly deferred rather than attempted.