# FINISHER — Complete Reference (flattened) > The .skill bundle flattened into one file for reading, pasting, or uploading as a > knowledge file to ChatGPT / Gemini. The bundle is the version to actually install in a > coding tool, because it loads only the sections a task needs and ships the scoring engine. --- --- name: finisher description: Turn a working prototype or vibe-coded app into a production-ready product, and score it. Audits 168 checks across 19 domains - auth and authorization, secrets, data and migrations, API correctness, frontend and accessibility, application security, supply chain, CI/CD, environments, observability, reliability, performance, cost and abuse control, AI and agent safety, privacy and compliance, testing, mobile and app store, handoff docs, product outcomes - and produces a 0-100 FINISHER Score with a pre-score, post-score, and change log every run so progress is measurable. Use when asked whether an app is production ready, safe, secure, launchable, scalable, or handoff ready; before launching, adding payments or auth, connecting a database, or shipping to an app store; when auditing or hardening AI-generated code; or when debugging exposed keys, runaway API bills, broken auth, missing logs, or mystery production failures. version: 2.0.0 --- # FINISHER **A working demo is not a finished product.** This skill closes that gap and measures the closing. Every run produces a **pre-score, a post-score, and a log of exactly what moved the number**. The score is computed deterministically from an auditable state file, so the same project state yields the same number regardless of who runs it — which is what makes progress across runs comparable rather than anecdotal. --- ## Operating principles 1. **Assess before building.** No new features until the current risks are known. 2. **Evidence over assertion.** A control you have not demonstrated is not a control. The scoring scale enforces this. 3. **Authentication is not authorization.** Login proves identity. Permissions prove access. 4. **Trust no generated code until reviewed.** AI code can work while being insecure, expensive, and unmaintainable. Measured data says roughly 45% of AI-generated code contains a vulnerability, and that number has not improved in two years. 5. **Secrets never belong in client code, git history, logs, screenshots, or AI prompts.** Rotate anything ever exposed. Deleting is not rotating. 6. **Instructions are not a security control.** This applies to prompts given to agents and to comments telling humans not to do something. If the capability exists, assume it will be used. 7. **If it breaks and nothing records it, it is not production-ready.** 8. **Every paid endpoint needs a budget boundary** before launch, not after the bill. 9. **Every database change is a migration. Every deployment is reversible. Every backup has been restored.** 10. **Prefer boring, battle-tested services for dangerous problems.** Do not build auth, payments, crypto, or secret management. 11. **Score honestly.** When torn between two scores, take the lower one. The goal is finding work, not looking finished. 12. **Verify volatile facts.** Regulations, store policies, tool flags, and provider pricing change faster than any document. Where this skill says "verify," search primary sources rather than asserting from memory. --- ## The run loop ### 1. Intake Read the repository before asking questions — see `references/01-intake-and-tiering.md`. Then write `finisher/context.json`: ```json {"has_users":true,"is_public":true,"is_multi_tenant":false,"has_admin":true, "has_pii":true,"has_payments":true,"has_uploads":false,"has_ugc":true, "has_ai":true,"has_agents":false,"has_third_party":true,"has_jobs":true, "has_email":true,"has_mobile":false,"eu_users":true,"us_state_privacy":true, "is_regulated":false,"has_team":true} ``` These conditions decide which checks apply. **When unsure, set true** — a false negative silently hides a P0. Assign a risk level (L0 local demo through L4 enterprise/regulated) and state the scope of this run: triage, full audit, domain deep-dive, regression, or pre-launch gate. ### 2. Initialize (first run only) ```bash python3 scripts/finisher_score.py init \ --context finisher/context.json --project "myapp" -o finisher/state.json ``` ### 3. Assess Work through the applicable checks in `assets/checks.yaml`, consulting the domain reference for depth. For each check, record in `state.json`: - `score` 0–4 - `evidence` — what you observed (required at 2+) - `artifact` — file path, test name, command output, or URL a stranger could re-run (required at 3+) - `enforcement` — the CI gate, runtime policy, or alert that prevents regression (required at 4) Score from what you verified, not from what the code appears to do. "The code calls `rateLimit()`" is a 1. Hitting the endpoint 200 times and seeing 429s is a 3. ### 4. Fix, in order 1. Stop feature work · 2. Branch and back up · 3. Rotate exposed secrets · 4. Fix authorization 5. Separate environments · 6. Migrations, backups, and a real restore · 7. Dependency and secret scanning 8. Error tracking, structured logs, uptime monitoring · 9. Rate limits and spend caps 10. Retries, idempotency, background jobs · 11. Core tests and edge-case QA 12. Preview deploys, tested rollback, feature flags · 13. Privacy, deletion, legal basics 14. Load test · 15. Handoff docs · 16. Private beta · 17. Measure real outcomes Update `state.json` as each check moves. ### 5. Commit the run ```bash python3 scripts/finisher_score.py validate # anti-gaming gate; must pass python3 scripts/finisher_score.py commit \ --run-id $(date +%Y-%m-%d)-r1 \ --actions finisher/actions.json \ --commit-sha $(git rev-parse --short HEAD) ``` `actions.json` maps check IDs to what you actually did, plus a summary: ```json {"summary":"Closed both auth P0s and verified the restore path.", "actions":{ "AUTHZ-01":"Moved ownership filtering into the repository layer; added 14 IDOR cases.", "DATA-02":"Restored yesterday's snapshot into a scratch project, booted the app, verified row counts. 11 minutes — that is our real RTO."}} ``` This writes `finisher/SCORE.md`, `finisher/runs/.md` (pre, post, delta, change log), and appends to `finisher/ledger.jsonl`. ### 6. Report Lead with the three numbers. Then the band, the open P0 list, and the highest-value targets for the next run. Never bury a P0 under good news. --- ## Scoring, in brief Full specification: **`references/00-scoring-system.md`** — read it before your first run. **Scale (per check):** 0 absent · 1 claimed · 2 implemented · 3 verified · 4 enforced **Ceilings — the rule that keeps the score honest:** | Condition | Composite capped at | |---|---| | Any P0 open | **39** | | Any P1 open | **69** | | Any P2 open | **89** | A project with excellent tests, great observability, and one exposed API key scores 39. That is correct. **Bands:** 0–39 do not launch · 40–59 private beta · 60–74 paid beta with fixes · 75–89 production ready · 90–100 scale ready **Coverage** is reported alongside the score. Below 80%, the number describes what you looked at rather than what is true, and should not be quoted to anyone making a launch decision. --- ## Domain map | ID | Domain | Wt | Reference | Applies when | |---|---|---:|---|---| | D01 | Identity, Auth & Authorization | 12 | `references/02-auth-and-authorization.md` | has_users | | D02 | Secrets & Key Management | 8 | `references/03-secrets-and-keys.md` | always | | D03 | Data Layer, Migrations & Backups | 9 | `references/04-data-and-migrations.md` | always | | D04 | API & Backend Correctness | 7 | `references/05-api-and-backend.md` | always | | D05 | Frontend, UX & Accessibility | 6 | `references/06-frontend-ux-a11y.md` | always | | D06 | Application Security | 9 | `references/07-appsec.md` | always | | D07 | Supply Chain & Dependency Integrity | 6 | `references/08-supply-chain.md` | always | | D08 | CI/CD & Release Engineering | 6 | `references/09-cicd-and-release.md` | always | | D09 | Environments, Config & Infrastructure | 5 | `references/10-environments-and-infra.md` | always | | D10 | Observability & Error Tracking | 7 | `references/11-observability.md` | always | | D11 | Reliability, Backup & Incident Response | 7 | `references/12-reliability-and-incidents.md` | always | | D12 | Performance, Caching & Scale | 5 | `references/13-performance-and-scale.md` | always | | D13 | Cost Control & Abuse Prevention | 5 | `references/14-cost-and-abuse.md` | always | | D14 | AI & Agent Safety and Economics | 6 | `references/15-ai-and-agents.md` | has_ai | | D15 | Privacy, Legal & Compliance | 6 | `references/16-privacy-legal-compliance.md` | has_pii | | D16 | Testing & Verification | 6 | `references/17-testing-and-verification.md` | always | | D17 | Mobile & App Store | 6 | `references/18-mobile-and-app-store.md` | has_mobile | | D18 | Documentation & Handoff | 4 | `references/19-handoff-and-docs.md` | always | | D19 | Product Truth & Outcome Measurement | 3 | `references/19-handoff-and-docs.md` | always | Cross-cutting: `references/20-manual-test-scripts.md` (the scripts scanners cannot replace), `references/21-ai-code-failure-modes.md` (the review checklist and the evidence behind it), `references/22-command-appendix.md` (copy-paste commands). --- ## Triage: the P0 set If you have one hour, check these. Every one is a launch blocker on its own. | Check | Question | |---|---| | SEC-01 | Is any secret reachable from the client bundle? | | SEC-02 | Does a secret scan of full git history pass, and was anything exposed rotated? | | AUTH-01 | Is authentication hand-rolled? | | AUTH-02 | Does every protected API endpoint enforce auth server-side, not just the UI route? | | AUTHZ-01 | Can user A reach user B's data by changing an ID anywhere? | | AUTHZ-02 | Is tenant isolation enforced at the database layer? | | AUTHZ-03 | Can admin access be obtained client-side? | | AUTH-03 / AUTH-05 | Do sessions expire, does logout invalidate, are reset tokens single-use? | | API-01 | Are price, role, and ownership recomputed server-side? | | APPSEC-01 / 02 | Injection prevented structurally? Output encoded in every context? | | DATA-01 / 02 | Do backups exist, and has a restore actually been performed? | | DATA-03 / 04 | Are environments separate? Does anyone or any agent hold production write access? | | SEC-08 | Are payment webhooks signature-verified and replay-safe? | | SUP-01 | Do install scripts run for arbitrary transitive dependencies? | | CI-01 / CI-05 | Is it in version control? Has rollback been tested? | | ENV-01 | Are dev, staging, and production genuinely separate? | | OBS-01 | Does production error tracking exist and receive errors? | | COST-01 / 02 | Is any paid endpoint public and unmetered? Are spend caps set? | | AI-01 / 02 / 03 | Provider calls server-side only? Lethal trifecta broken? Any agent holding production credentials? | | LEG-01 / 02 | Privacy policy published and accurate? Data inventory exists? | --- ## Response style - **Lead with the score and the P0 list.** Never bury a blocker under good news. - **Never reassure without evidence.** "Looks fine" is not an assessment. - **Cite the check ID** so findings map to the score. - **Give the command, not the concept.** "Add rate limiting" is not actionable. The middleware configuration is. - **Label unknowns explicitly.** Unassessed is not the same as passing, and coverage makes the difference visible. - **Do not invent versions, pricing, regulatory dates, or store policies.** Verify or say you have not. - **Do not recommend enterprise tooling when a simple control closes the risk.** - **Do not slow down harmless prototypes.** An L0 project with no users, no real data, and no secrets in git is finished. Say so and stop. - **Never print a secret value**, even one you found. Report the location and the rotation step. --- ## Bundle contents ``` SKILL.md this file assets/checks.yaml 168 checks: id, domain, tier, weight, applicability, risk, fix, evidence, enforcement assets/state.template.json shape of the state file assets/context.template.json intake conditions assets/templates/ RUNBOOK, ARCHITECTURE, ADR, THREAT-MODEL, INCIDENT-POSTMORTEM, README, SCORECARD, run report scripts/finisher_score.py scoring engine: init, validate, score, commit, history (no required dependencies) references/00..22 the depth behind each domain ``` Extend the catalog freely — stack-specific and organization-specific checks belong in `checks.yaml`. Bump `meta.catalog_version` when you do, so score comparisons across versions stay interpretable. --- # The FINISHER Score The score exists to answer one question honestly: **is this closer to shippable than it was last time, and by how much?** Everything here is designed so that the same project state produces the same number regardless of who or what runs the assessment. A score you can argue with is a score you can move. --- ## 1. The three numbers Every run produces three, and reporting fewer than three is a defect. | Number | What it means | Where it comes from | |---|---|---| | **Pre-score** | The score at the start of this run | The previous ledger entry. Never entered by hand. | | **Post-score** | The score after this run's work | Computed from the updated state file | | **Delta + change log** | Which specific checks moved and what was done to move them | Diffed automatically; the "what was done" text comes from `actions.json` | The first run has no pre-score. Say `n/a (first run)` — do not invent a baseline. --- ## 2. The measurement scale Every check is scored 0–4. The scale is about **proof**, not effort. | Score | Level | What it means | What is required to claim it | |---:|---|---|---| | 0 | **ABSENT** | Not present, or you looked and could not find it | Nothing | | 1 | **CLAIMED** | Something exists. Nobody has demonstrated it works. | A pointer to where it supposedly lives | | 2 | **IMPLEMENTED** | It works. You saw it work once. | `evidence`: a sentence describing what you observed | | 3 | **VERIFIED** | Proven by something a stranger could re-run | `artifact`: a file path, test name, command output, screenshot, or URL | | 4 | **ENFORCED** | Automated so it cannot silently regress | `enforcement`: the CI job, runtime policy, or alert that holds the line | Three rules that make the scale mean something: 1. **You cannot score 2+ without evidence.** The validator rejects it. 2. **You cannot score 3+ without a named artifact.** "We tested it" is a 2. "`tests/authz.spec.ts::cross-tenant-read` passes" is a 3. 3. **You cannot score 4 without naming what enforces it.** A thing a human remembers to do is a 3 forever. The gap between 3 and 4 is where most projects quietly rot. A verified control with no enforcement point degrades to absent within about two months of active development. ### The distinction that matters most **2 is "it works." 3 is "it is proven to work." 4 is "it cannot stop working without someone noticing."** AI-built codebases are unusually good at producing 1s and unusually bad at producing 4s. The whole point of this exercise is to move the P0 set to 3+ and the highest-risk controls to 4. --- ## 3. Priority tiers and the ceiling rule | Tier | Meaning | Counts as closed at | |---|---|---:| | **P0** | Breach, data loss, outage, money loss, or legal exposure. Blocks any launch. | 3 | | **P1** | Stability, trust, maintainability, support burden. Blocks paid or public launch. | 3 | | **P2** | Scale, polish, efficiency, long-horizon risk. Blocks confident growth. | 2 | **The ceiling rule is what makes the score honest.** Weighted averages let you offset a catastrophe with polish. This one does not. ``` any P0 open -> composite capped at 39 (DO NOT LAUNCH) any P1 open -> composite capped at 69 (cannot exceed PAID BETA WITH FIXES) any P2 open -> composite capped at 89 (cannot claim SCALE READY) ``` A project with beautiful observability, exhaustive tests, and one exposed API key scores 39. That is correct. The number should refuse to flatter you. The scorecard always shows both figures: the **uncapped weighted score** (how much work is genuinely done) and the **capped composite** (what you are allowed to claim). Watching the uncapped number rise while the capped number sits at 39 is a useful, uncomfortable signal that you are polishing instead of unblocking. --- ## 4. Readiness bands | Score | Band | What it licenses | |---:|---|---| | 0–39 | **DO NOT LAUNCH** | Internal use with fake data only | | 40–59 | **PRIVATE BETA ONLY** | Invited users who know it is early, no sensitive data, no money | | 60–74 | **PAID BETA WITH FIXES** | Real users and money, with disclosed limitations and a fix list | | 75–89 | **PRODUCTION READY** | Public launch, support commitment, sleep at night | | 90–100 | **SCALE READY** | Growth, enterprise conversations, on-call rotation | Bands are permissions, not grades. 78 does not mean "B+". It means you may launch. --- ## 5. How the composite is computed ``` check_points(c) = weight(c) x score(c) # score 0-4, weight 1-5 domain_score(d) = 100 x sum(check_points) / sum(weight x 4) # over applicable checks raw_composite = sum(domain_weight x domain_score) / sum(domain_weight) composite = min(raw_composite, tier_ceiling) ``` **Applicability, not zeroes.** A check that does not apply — no payments, no mobile app, no AI — is excluded from both numerator and denominator. It is not scored 0. A static marketing site is not penalized for lacking tenant isolation. Domain weights are also conditional. `D14 AI & Agent Safety` carries weight 6; if the project has no AI, the domain drops entirely and the remaining weights renormalize. There is no manual reweighting to do. **Assessment coverage** is reported alongside the score: ``` coverage = assessed applicable checks / total applicable checks ``` This is the single most important guard against a misleading score. A 45 with 96% coverage means *we looked hard and it is rough*. A 45 with 30% coverage means *we barely looked*. Below 80% coverage, the scorecard prints a warning and the number should not be quoted to anyone making a decision. --- ## 6. The state file `finisher/state.json` is the only place scores live. One entry per check: ```json { "AUTHZ-01": { "score": 3, "evidence": "Ran the User A / User B script against staging. All 14 ID-substitution attempts returned 403 and were logged.", "artifact": "tests/security/authz.spec.ts::cross-user-object-access (14 cases, passing) + docs/evidence/2026-07-25-authz-run.txt", "enforcement": null, "notes": "Export endpoint was the one that failed initially; fixed in a1b2c3d." } } ``` Fields: - `score` — 0–4. - `evidence` — what you observed. Required at 2+. - `artifact` — what a stranger could re-run or open. Required at 3+. - `enforcement` — the automation that prevents regression. Required at 4. - `na` / `na_reason` — excludes the check. **A reason is mandatory**, and the validator warns if you mark something N/A that the intake context says applies. This is the main gaming vector; it is deliberately noisy. - `notes` — anything a future reader needs. --- ## 7. The run loop ```bash # once, at the start of the project python3 scripts/finisher_score.py init --context finisher/context.json --project "myapp" \ -o finisher/state.json # every run, after doing the work and updating state.json python3 scripts/finisher_score.py validate # anti-gaming gate python3 scripts/finisher_score.py commit \ --run-id 2026-07-25-r1 \ --actions finisher/actions.json \ --commit-sha $(git rev-parse --short HEAD) python3 scripts/finisher_score.py history # the trend ``` `commit` writes three things: | File | Purpose | |---|---| | `finisher/SCORE.md` | Current scorecard. Overwritten each run. This is the file you show people. | | `finisher/runs/.md` | This run's pre/post/delta and change log. Never overwritten. | | `finisher/ledger.jsonl` | Append-only history: one JSON line per run with every check score. The audit trail. | `commit` **refuses to run on an invalid state** unless forced. That is intentional — an unvalidated score is worse than no score. ### actions.json The change log. Keys are check IDs; values describe what was actually done. ```json { "summary": "Closed the two auth P0s and got backups verified. Cost controls next.", "actions": { "AUTHZ-01": "Moved ownership filtering into the repository layer so an unscoped query is unrepresentable; added 14 IDOR cases.", "DATA-02": "Restored the 2026-07-24 snapshot into a scratch project, booted the app against it, verified row counts. 11 minutes end to end — that is our real RTO.", "SEC-02": "gitleaks found a Stripe test key in commit 4f2a1c from March. Rotated at the provider even though it was test mode." } } ``` Any action key that does not correspond to a score change is reported separately under **"Work done that did not move a score."** Do not delete those entries. Work that did not move the number is signal: either it was not finishing work, or the check needs evidence you have not captured yet, or you are working on something the catalog does not measure — and that last case means the catalog needs a check. --- ## 8. Reading the trend `history` gives the trajectory: ``` run score delta P0 P1 cov% band 2026-07-25-r1 3.1 +3.1 26 88 2.5 DO NOT LAUNCH 2026-07-26-r2 12.8 +9.7 19 88 18.9 DO NOT LAUNCH 2026-07-29-r3 31.0 +18.2 6 84 61.4 DO NOT LAUNCH 2026-08-02-r4 54.2 +23.2 0 71 88.1 PRIVATE BETA ONLY ``` What to read from it: - **Early runs should mostly raise coverage, not score.** Run 1 is an audit; a big score jump in run 1 means someone scored from optimism rather than evidence. - **The jump when the last P0 closes is the ceiling lifting**, not a sudden burst of quality. Expect it and do not over-celebrate it. - **A flat score with high activity** means work is going into features, not finishing. That is a legitimate choice — but now it is visible. - **Any negative delta must have an explanation in the run report.** An unexplained regression is itself a finding: something is being undone faster than it is being built, usually by an agent that does not know the constraint exists. Two derived metrics worth watching: - **Points per run** — your finishing velocity. Use it to estimate runs-to-band. - **Regression rate** — checks that went down over the last five runs. Above ~5% of moved checks, your problem is not building controls, it is holding them. Push the highest-regression controls from 3 to 4. --- ## 9. Anti-gaming rules The score is only useful if it is hard to fake. The validator enforces the mechanical rules; these are the judgement ones. 1. **Never score from the code alone when the check asks about runtime.** "The code calls `rateLimit()`" is a 1. Hitting the endpoint 200 times and seeing 429s is a 3. 2. **Never score a check higher because a related check is strong.** Backups configured (DATA-01) does not lift restore-tested (DATA-02). They are separate checks precisely because teams conflate them. 3. **N/A requires a reason, and the reason must be about the product, not the schedule.** "No payments in this product" is valid. "Not doing payments yet" means the check is in scope and scores 0. 4. **A check regresses when its evidence goes stale.** A restore drill from eight months ago is not a current 3. Downscore it, and let the regression show in the log. 5. **When uncertain between two scores, take the lower one.** The purpose is to find work, not to look finished. 6. **Do not add checks to raise the denominator.** If you extend the catalog, add checks in the catalog file with weights and tiers, and note the catalog version bump in the run report so historical comparisons stay interpretable. --- ## 10. Extending the catalog `assets/checks.yaml` is the source of truth. To add a check: ```yaml - id: ORG-01 # unique, domain-prefixed domain: D06 # existing domain, or add one with a weight tier: P1 # P0 / P1 / P2 weight: 3 # 1-5, relative to other checks in the domain when: [has_users, is_public] # ANDed; `always` for universal; `!cond` to negate title: "One sentence, stated as the thing being true" risk: "What goes wrong if it is not true, ideally with a concrete failure" fix: "What to actually do" evidence: "What proves score 3" enforce: "What proves score 4" ``` Bump `meta.catalog_version` when you change scoring-relevant fields. The ledger records the catalog version per run, so a score comparison across a version change is flagged rather than silently misleading. Stack-specific and organization-specific checks belong here. A catalog that never grows is a catalog nobody is using. --- # Intake, Risk Tiering and Scoping The first ten minutes of a FINISHER run decide whether the rest of it is useful. Get the context wrong and you will either terrorize someone's weekend project or wave through a payments app. ## 1. Do not ask; look first Read before you interview. The repository answers most intake questions faster and more honestly than the owner will. ```bash # shape of the thing ls -la; cat README* 2>/dev/null | head -60 cat package.json requirements.txt pyproject.toml go.mod Gemfile composer.json 2>/dev/null # what it talks to grep -rEl "stripe|paddle|lemonsqueez|openai|anthropic|gemini|supabase|firebase|clerk|auth0|twilio|sendgrid|resend|s3|cloudinary" \ --include="*.{ts,tsx,js,jsx,py,go,rb,java,php}" . 2>/dev/null | head -40 # secret names only, never values grep -rhoE "(process\.env\.|import\.meta\.env\.|os\.environ\[?.?)[A-Z_][A-Z0-9_]*" . 2>/dev/null \ | grep -oE "[A-Z_][A-Z0-9_]*$" | sort -u # surface area find . -path ./node_modules -prune -o \( -name "*route*" -o -name "*controller*" -o -name "*api*" \) -print 2>/dev/null | head -40 ls migrations/ prisma/migrations/ supabase/migrations/ db/migrate/ alembic/versions/ 2>/dev/null ls .github/workflows/ 2>/dev/null ``` Then ask only what the code cannot tell you: user counts, launch date, who the users are, what data is genuinely sensitive, and what the owner is actually afraid of. ## 2. Build the context object Everything downstream keys off this. Write it to `finisher/context.json`. ```json { "has_users": true, "is_public": true, "is_multi_tenant": false, "has_admin": true, "has_pii": true, "has_payments": true, "has_uploads": false, "has_ugc": true, "has_ai": true, "has_agents": false, "has_third_party": true, "has_jobs": true, "has_email": true, "has_mobile": false, "eu_users": true, "us_state_privacy": true, "is_regulated": false, "has_team": true } ``` Getting these wrong is the most consequential error in the whole process, because a false condition silently removes checks from the denominator. Two rules: - **When unsure, set it true.** A false positive costs a conversation. A false negative hides a P0. - **`has_pii` is true more often than people think.** Email addresses are personal data. IP addresses in logs are personal data. "We don't collect PII, just accounts" is wrong. ## 3. Risk levels The level sets expectations, not the check list. The check list is set by the context conditions. The level tells you how hard to push and what "done" means for this project right now. | Level | Shape | Target band | What a FINISHER run should do | |---|---|---|---| | **L0** Local demo | No users, no real data, no public URL | n/a | Confirm no secrets committed, confirm it is genuinely not exposed, then stop. Do not run the full catalog. | | **L1** Private prototype | Invited users, low-risk data, no money | 40+ | P0s only, plus error tracking and backups. A weekend of work. | | **L2** Paid beta | Real users, real money, real data | 60+ | All P0, most P1. Staging, CI, monitoring, restore test, spend caps. | | **L3** Production SaaS | Public, private data, support expectation | 75+ | All P0 and P1. Runbook, rate limits, security scanning, handoff docs. | | **L4** Enterprise / regulated | Compliance, large orgs, sensitive data | 90+ | Everything, plus audit logging, access reviews, formal compliance mapping, vendor risk. | Choosing a level is a business decision, not a technical one. Ask directly: *"if this leaked every user's data tomorrow, what happens?"* The honest answer sets the level faster than any questionnaire. **The one rule that overrides the level:** if the product touches money, personal data, or production systems belonging to someone else, it is at least L2 regardless of user count. Ten users is not a safety property. ## 4. Scoping a run A full first pass on a real application is genuinely hours of work. Scope deliberately: | Run type | Scope | Typical duration | |---|---|---| | **Triage** | P0 checks only, all domains | 1–2 hours | | **Full audit** | Every applicable check, evidence gathered | half a day to two days | | **Domain deep-dive** | One domain, every check, to level 3–4 | 2–4 hours | | **Regression run** | Re-verify checks at 3+ whose evidence is over 90 days old | 1 hour | | **Pre-launch gate** | Every P0 and P1, evidence refreshed | half a day | Always record the scope in the run report's summary. A score computed from a triage run is not comparable to one from a full audit — coverage percentage makes that visible, which is exactly why coverage is reported. ## 5. What to do first, always Regardless of level, in this order: 1. **Stop feature work.** Not forever. Until the P0 list is known. You cannot assess a moving target, and every new feature added during an audit is unassessed surface. 2. **Create a branch and take a backup.** Before touching anything. Especially the database. 3. **Run the secrets check.** It is the only P0 that gets worse with every hour of delay, because exposure time is the variable. 4. **Establish the score baseline** with honest coverage, even if the honest answer is "we have looked at 12% of this." ## 6. When to say the project is fine FINISHER is a tool for protecting people, not a ritual. A weekend project with no users, no real data, no public URL, and no secrets in git is **finished**. Say so, record it as L0, and give the owner the two-line list of what would change if they ever put it online. Applying L3 discipline to an L0 project wastes the owner's time and trains them to ignore you when it matters. --- # Auth and Authorization > Covers checks AUTH-01..12, AUTHZ-01..04. Domain D01, weight 12 — the heaviest domain in the catalog. **Authentication proves who you are. Authorization decides what you may touch.** AI-generated applications get the first right and the second wrong with startling consistency. Broken Access Control leads the OWASP Top 10 in the 2025 edition, and Missing Authorization sits at #4 on the 2025 CWE Top 25. The failure is structural, not careless. A model asked to "add a dashboard route" produces a handler that fetches by ID. Nothing in that request says "and check the caller owns it," and nothing in the resulting code reveals the omission. It works perfectly in a single-user demo. --- ## 1. Do not build authentication Use a battle-tested provider or your framework's first-party auth. The list of things you have to get exactly right — password hashing parameters, timing-safe comparison, session fixation, reset token entropy and single-use semantics, enumeration resistance, credential-stuffing defense — has no partial credit. Reasonable choices in 2026: Clerk, Auth0, WorkOS, Supabase Auth, Firebase Auth, Cognito, Better Auth, Auth.js/NextAuth with a vetted adapter, Devise, Django's auth, Rails' `has_secure_password`. If custom auth already exists in the codebase, that is a P0 finding. Migrating is usually a day of work and removes an entire category of risk permanently. ## 2. The authorization pattern that actually holds Controller-level checks rot. Someone adds an endpoint, forgets the guard, and there is no signal. Push the constraint down until an unscoped query is **unrepresentable**: ```ts // Fragile: the check is a line of code someone can forget const doc = await db.document.findUnique({ where: { id } }); if (doc.ownerId !== session.userId) throw forbidden(); // Better: scoping is part of how you ask for data const doc = await forUser(session).document.findUnique({ where: { id } }); // Best: the database refuses regardless of what the application asks // Postgres RLS: policy USING (tenant_id = current_setting('app.tenant_id')::uuid) ``` Three layers, in ascending order of durability: 1. **Route middleware** — is there a session at all 2. **Data-access scoping** — every query carries the actor's identity 3. **Database policy (RLS or per-tenant schema)** — the backstop that survives an application bug For multi-tenant products, layer 3 is not optional. One missing `WHERE tenant_id = ?` is a full customer-data breach, and that is a single-token mistake an AI makes routinely. ### Row-level security, concretely ```sql ALTER TABLE documents ENABLE ROW LEVEL SECURITY; ALTER TABLE documents FORCE ROW LEVEL SECURITY; -- applies to the table owner too CREATE POLICY tenant_isolation ON documents USING (tenant_id = current_setting('app.current_tenant', true)::uuid) WITH CHECK (tenant_id = current_setting('app.current_tenant', true)::uuid); ``` Three traps: `USING` alone protects reads but not writes — you need `WITH CHECK` too; policies do not apply to the table owner unless you `FORCE`; and a connection pooler can leak `current_setting` between requests unless you set it inside the transaction. Test all three. A CI test worth having: ```sql -- fails if any table with a tenant_id column lacks RLS SELECT c.relname FROM pg_class c JOIN pg_attribute a ON a.attrelid = c.oid AND a.attname = 'tenant_id' WHERE c.relkind = 'r' AND NOT c.relrowsecurity; ``` ## 3. The route inventory You cannot secure what you have not enumerated. Build this table and keep it current; it is the artifact that lifts AUTH-02 to level 3. | Method | Path | Public? | Authn | Authz rule | Rate limited | Test | |---|---|---|---|---|---|---| | GET | /api/documents/:id | no | session | owner or org member with `doc:read` | yes | `authz.spec.ts::doc-read` | | POST | /api/admin/users | no | session | role=admin, MFA required | yes | `admin.spec.ts::create-user` | | GET | /api/health | yes | none | none | yes | `health.spec.ts` | Generate it from the router where you can. The valuable column is **Authz rule** — writing it forces you to notice the endpoints where you cannot state one. ## 4. Sessions and tokens - **Regenerate the session identifier immediately after login and after any privilege change.** Skipping this is session fixation. - **Logout must invalidate server-side.** Deleting a client cookie while the token stays valid is theatre. Test by capturing a token, logging out, and replaying it. - **Cookies:** `Secure; HttpOnly; SameSite=Lax` at minimum, `Strict` where the UX allows. Use the `__Host-` prefix (which requires `Secure`, `Path=/`, and no `Domain` attribute) for the strongest binding. - **Idle and absolute timeouts, both enforced server-side.** A client-side timer is a UX feature, not a control. - **JWTs:** pin the expected algorithm at verification — `alg: none` and algorithm-confusion attacks persist because libraries trust the token's own header. Keep access tokens short-lived with rotating refresh tokens, or use server-side sessions. Remember JWTs have no native revocation: if you need instant logout, you need a server-side lookup or a denylist. - **Never put a token in a URL.** It lands in logs, referrer headers, and browser history. ## 5. Password reset — the most-broken flow in AI-generated code Requirements, all of which are commonly missing: - Cryptographically random token, **hashed at rest** (a leaked database should not yield working reset links) - Expires in 15–60 minutes - **Single use** — invalidated the moment it is consumed - Invalidated when the password changes by any other route - **All existing sessions invalidated** on password change - Response is identical whether or not the account exists (enumeration) - Rate limited per account and per IP Test every one of these. The reuse test — click the same reset link twice — catches the most common defect in about fifteen seconds. ## 6. Rate limiting the auth surface Lock on **account identity**, not IP alone; attackers rotate IPs and a shared corporate NAT means IP locking punishes innocents. Combine per-identity throttling with per-IP limits, add exponential backoff, and keep error text generic. Add a breached-password check at registration and at password change — credential stuffing is the actual attack, and blocking known-compromised passwords defeats most of it. ## 7. Admin surfaces Treat admin as a separate application that happens to share a database: - Separate middleware with its own test suite - Role verified server-side on every request, never from a client-supplied claim - MFA mandatory - Consider a separate hostname, IP allowlist, or SSO requirement - Every admin action written to an append-only audit log with actor, target, before/after, and request ID The audit log is what turns "we think nobody misused it" into an answer. ## 8. What to actually test Run the full script in `20-manual-test-scripts.md`, then automate it. The automated version is the highest-value test suite in an AI-built codebase: ```ts // The shape that matters: enumerate endpoints, enumerate actors, assert rejection for (const ep of PROTECTED_ENDPOINTS) { for (const actor of [anonymous, otherUser, otherTenant, lowerRole]) { it(`${ep.method} ${ep.path} rejects ${actor.name}`, async () => { const res = await call(ep, actor); expect([401, 403, 404]).toContain(res.status); expect(res.body).not.toMatchObject({ id: ep.fixtureId }); // no partial leak }); } } ``` Make the endpoint list derive from the router, not a hand-maintained array. Then adding an unprotected route fails the build — which is what moves AUTH-02 and TEST-01 from 3 to 4. ## 9. Common findings, in the order you will find them 1. API route protected in the UI but open at the JSON endpoint 2. Ownership checked on read, forgotten on update, delete, or export 3. Tenant filter present in the list query, absent in the detail query 4. Admin flag read from a client-supplied JWT claim the client can edit 5. Reset token that never expires or works more than once 6. Logout that only clears the cookie 7. Bulk or export endpoints that skip the scoping the single-record endpoint applies 8. A "public" share link that grants more than the sharer had 9. Webhook and callback endpoints with no authentication at all 10. GraphQL resolvers or ORM includes that traverse to unauthorized related records --- # Secrets and Key Management > Covers checks SEC-01..08. Domain D02, weight 8. **A secret that has ever been in a repository, a client bundle, a log, a screenshot, or an AI prompt is compromised.** Not "at risk" — compromised. Deleting it does not help. Rotation is the only remediation, and it is cheap. Do it. --- ## 1. The client-side rule Anything shipped to a browser or a mobile binary is public. Not obscured — public. The framework prefixes exist to make this explicit: ``` NEXT_PUBLIC_* VITE_* REACT_APP_* EXPO_PUBLIC_* PUBLIC_* ``` Every one of those is compiled into the bundle. If a service key ever ends up behind one of those prefixes, it is on the internet. The check: ```bash npm run build # adjust the output dir to your framework grep -rohE "(sk_live_|sk_test_|rk_live_|xox[baprs]-|ghp_|github_pat_|AKIA[0-9A-Z]{16}|AIza[0-9A-Za-z_-]{35}|SG\.[A-Za-z0-9_-]{20,}|-----BEGIN [A-Z ]*PRIVATE KEY-----)" \ dist/ build/ .next/static/ .output/ 2>/dev/null | sort -u # expect: nothing # then the manual pass grep -riE "(secret|api_?key|password|token|credential)" dist/ .next/static/ 2>/dev/null | head -30 ``` Then open DevTools on the running app, look at the Sources tab, and read the Network tab for outbound requests carrying an `Authorization` header your server did not add. **Publishable keys are fine.** Stripe publishable keys, Supabase anon keys, Firebase web config, PostHog project keys — these are designed for client exposure. What matters is that the *secret* counterpart never appears, and that anon-key access is constrained by policy (RLS, security rules) rather than by nobody noticing. ## 2. Git history ```bash gitleaks git -v . # whole history gitleaks dir --redact --report-format sarif --report-path gitleaks.sarif . trufflehog git file://. --results=verified # verifies live credentials against providers git log --all --full-history -- .env .env.local .env.production find . -name "*.env*" -not -path "*/node_modules/*" ``` `trufflehog --results=verified` is worth the extra run: it actually calls the provider to check whether a found credential is still live, which turns a wall of maybes into a short list of emergencies. **On a hit:** rotate first, then decide about history. Rewriting history with `git filter-repo` or BFG is disruptive and does not help if the repository was ever cloned, forked, or mirrored. Rotation is what actually closes the exposure. ## 3. Where secrets should live In ascending order of maturity: 1. Platform environment variables, scoped per environment (fine for most projects) 2. A managed secret manager — AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, Doppler, Infisical 3. **Short-lived credentials generated per session or deployment**, so a stolen credential expires on its own Level 3 is the direction of travel and the reason OIDC federation matters: the best rotation policy is not needing one. Non-negotiables at any level: - `.env` in `.gitignore`; `.env.example` lists **names only** - Different values per environment — no shared keys between staging and production - Production secrets not present on any developer machine - Access to the secret store is itself restricted and logged ## 4. CI/CD: stop using long-lived cloud keys A static cloud access key in CI is a permanent credential that has been the payload of every major registry worm of the last two years. Replace it with OIDC federation. ```yaml permissions: id-token: write # without this, no token is minted contents: read jobs: deploy: runs-on: ubuntu-latest steps: - uses: aws-actions/configure-aws-credentials@ # pin the SHA with: role-to-assume: arn:aws:iam:::role/deploy aws-region: us-east-1 ``` The trust policy on the cloud side should be scoped to repository **and** branch or environment, e.g. subject `repo:org/repo:environment:production`. A trust policy scoped only to the repository lets any branch — including one from a fork in some configurations — assume the role. Grant `id-token: write` only on the jobs that need it. Default the workflow to `contents: read`. ## 5. Keeping secrets out of logs and prompts Redaction belongs in the logger, not at the call sites: ```ts const REDACT = /^(authorization|cookie|set-cookie|x-api-key|password|token|secret|.*_key)$/i; function scrub(o: unknown, depth = 0): unknown { if (depth > 6 || o === null || typeof o !== "object") return o; return Object.fromEntries(Object.entries(o as Record) .map(([k, v]) => [k, REDACT.test(k) ? "[REDACTED]" : scrub(v, depth + 1)])); } ``` Also scrub: error-tracking payloads (request bodies and headers are attached by default in most SDKs), analytics events, and anything sent to a model provider. **Pasting a `.env` file into an AI chat is an exposure event** — treat it exactly like a commit. ## 6. Least privilege - Separate database roles for migrations, application reads, and application writes - Third-party API keys scoped to the minimum permission set, and IP-restricted where supported - Distinct keys per environment so a staging leak cannot touch production - Separate keys per service, so revoking one does not take down everything The test question: *for each credential, what is the worst thing its holder can do?* If the answer is "anything," it is over-privileged. ## 7. Rotation Rotation must be possible without downtime, or it will not happen during the incident when it matters. Support two valid credentials during rollover: add the new one, deploy, verify, remove the old one. Guidance has moved away from calendar-based rotation of user passwords toward rotation on suspected compromise — but **application secrets are different**. API keys and database credentials should rotate on a real cadence, and always immediately on: staff departure, vendor breach disclosure, any exposure, and any suspicion. Write the runbook, then rehearse it once while nothing is on fire. The rehearsal is what makes SEC-07 a 3. ## 8. Webhook signature verification Any endpoint a third party calls needs to prove the call came from them: - Verify the signature using the provider's library, against the **raw request body** — most frameworks parse JSON before you get to it, and re-serializing changes the bytes and breaks verification - Enforce a timestamp tolerance to reject replays - Store processed event IDs so a duplicate delivery produces one side effect - Return 2xx quickly and do the work asynchronously; providers retry on timeout, which is how you get duplicates An unverified payment webhook means anyone who can guess your URL can mark invoices paid. This is a P0 every time. --- # Data Layer, Migrations and Backups > Covers checks DATA-01..12. Domain D03, weight 9. Three facts drive this domain: an untested backup is a hypothesis; a schema change that is not a migration file will be lost; and an agent with production database credentials will eventually use them. --- ## 1. Backups and the restore drill **DATA-01 (backups configured) and DATA-02 (restore verified) are separate P0 checks on purpose.** Teams conflate them constantly. Configuration is a checkbox. Restoration is a fact. The drill, which is the artifact that makes DATA-02 a 3: 1. Take the most recent production snapshot 2. Restore it into a scratch environment — a new database, a branch, a fresh project 3. Point a running copy of the application at it 4. Verify: row counts on the three biggest tables, a specific known record, extensions and enums present, sequences correct 5. **Write down the elapsed time.** That number is your real RTO, and it is almost always several times what people guess. Cover every datastore, not just the primary: object storage buckets, search indexes, cache warm-up requirements, and any managed service holding state you cannot regenerate. Backup failures must alert. A backup job that has been silently failing for six weeks is the classic discovery-during-incident. ## 2. Environment separation Development, staging, and production get separate databases. Not separate schemas in one instance — separate instances or managed branches, with different credentials. The defensive pattern worth adding: ```ts // refuse to run destructive tooling against production const host = new URL(process.env.DATABASE_URL!).host; if (process.env.NODE_ENV !== "production" && /prod/i.test(host)) { throw new Error(`Refusing: non-production process pointed at ${host}`); } ``` Then: **no standing write access to production for humans or agents.** Read-only by default. Break-glass elevation that is time-boxed, logged, and alerts on use. AI coding tools get development credentials, full stop — the best-documented AI data-loss incident happened because an agent had production access at all, not because it chose to delete something. ## 3. Migrations Every schema change is a versioned file in the repository. No console DDL. The property to protect: **a fresh database, created from migrations alone, must produce a working application.** Verify it in CI: ```yaml - run: createdb ci_test - run: npm run migrate # from zero - run: npm run seed - run: npm test ``` ### Migration linting is the highest-value control here AI-written migrations take exclusive locks and drop data without hesitation. A linter catches both mechanically: ```bash npx squawk migrations/*.sql # Postgres: locks, missing CONCURRENTLY, unsafe types atlas migrate lint --dev-url "docker://postgres/16/dev" --latest 10 ``` Squawk flags the classics: creating an index without `CONCURRENTLY` (blocks writes for the duration), adding a unique constraint (exclusive lock, blocks everything), changing a column type (table rewrite), `serial` instead of identity, and adding a `NOT NULL` column without a safe default. Atlas adds destructive-change detection and drift. Make it a required PR check. It costs nothing and it catches the specific class of mistake that turns a routine deploy into an outage. ### Expand and contract Down migrations are a trap. They are rarely exercised, frequently wrong, and destructive down migrations lose data on exactly the worst day. Prefer roll-forward with backwards-compatible changes: 1. **Expand** — add the new column/table, nullable, no constraint 2. **Dual write** — deploy code that writes both old and new 3. **Backfill** — in batches, off the request path 4. **Switch reads** — deploy code that reads the new 5. **Contract** — a later deploy removes the old The property this buys you: **at every step, the previous version of the application still works against the current schema.** That is what makes a code rollback safe, which is the rollback you will actually perform. Test it explicitly — run version N-1's test suite against version N's schema. ## 4. Constraints and indexes AI-generated schemas routinely omit foreign keys, unique constraints, and `NOT NULL`, pushing integrity enforcement into application code that forgets it. The database is the only place a constraint cannot be bypassed by a new code path. Deliberate choices to make explicitly: - Foreign keys with chosen `ON DELETE` behavior — `CASCADE` is convenient and occasionally catastrophic - Unique constraints on natural keys and on idempotency keys - `NOT NULL` where a null genuinely has no meaning - Check constraints on enumerated values - Ownership columns (`user_id`, `tenant_id`) on every table holding user data, indexed, and non-nullable Index from measurement, not intuition: ```sql CREATE EXTENSION IF NOT EXISTS pg_stat_statements; SELECT calls, round(mean_exec_time::numeric,2) AS avg_ms, round(total_exec_time::numeric) AS total_ms, query FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20; -- indexes nobody uses (they cost write throughput) SELECT relname, indexrelname, idx_scan FROM pg_stat_user_indexes WHERE idx_scan = 0 ORDER BY relname; ``` Sort by `total_exec_time`, not `mean_exec_time` — the query that takes 30ms and runs a million times is the problem, not the one that takes 3 seconds once a day. ## 5. Idempotency Duplicate side effects are the signature failure of AI-generated handlers: double charges, duplicate orders, duplicate emails. The cause is always the same — retries, double clicks, and webhook re-delivery are normal, and nothing in the handler accounts for them. ```sql CREATE TABLE idempotency_keys ( key text PRIMARY KEY, scope text NOT NULL, response jsonb, created_at timestamptz NOT NULL DEFAULT now() ); ``` Insert the key inside the same transaction as the side effect. On conflict, return the stored response instead of doing the work again. Apply it to payments, outbound messages, external API calls with side effects, and anything a user can trigger twice by being impatient. Test it with genuine concurrency, not two sequential requests — the race is the interesting case. ## 6. File uploads Unrestricted file upload is CWE-434 and sits at #12 on the 2025 CWE Top 25. Extension checks are trivially bypassed. - Server-side size limit, enforced before buffering the whole file - Content-type determined by inspecting bytes, not by trusting the header or extension - Allowlist of types, never a denylist - Randomized stored filename; never use the client's filename on disk - Store in object storage with private ACLs; serve via short-lived signed URLs - Never serve uploads from a path that can execute - **SVG is an XSS vector** — sanitize it or serve it with `Content-Disposition: attachment` - Strip EXIF from images (it contains GPS coordinates) - For anything user-shared, virus scanning is worth the integration - Image and document processing libraries are a rich CVE source — run them out-of-process or in a sandbox ## 7. Retention and deletion Classify every table and bucket: what it holds, why, how long, who can read it. Then implement the purge job, and make deletion actually propagate — primary store, caches, search indexes, analytics, error tracking payloads, AI provider logs, and downstream processors. Backups are the honest hard case: a deleted record persists in backups until they expire. State the window, document it in your privacy notice, and make sure the window is finite. ## 8. Connection pooling — the first scaling wall Serverless plus a connection-per-invocation pattern exhausts Postgres connections at traffic levels far below what people expect. Do the arithmetic before launch: ``` max_concurrent_instances x connections_per_instance < db_max_connections - headroom ``` Use a pooler — PgBouncer, Supavisor, RDS Proxy, or your provider's built-in — in transaction mode for serverless. Note that transaction-mode pooling breaks prepared statements and session-level settings, which is a real constraint on how you set RLS context. Alert on pool saturation; it is the metric that predicts the outage rather than reporting it. --- # API and Backend Correctness > Covers checks API-01..09. Domain D04, weight 7. The backend is the only place a rule is real. Everything the client enforces is a suggestion that a determined user, a scripted client, or a modified mobile binary will decline. --- ## 1. Never trust the client, specifically The abstract version of this advice is useless. The concrete version is a list of things AI-generated handlers accept that they must not: | Client-supplied field | What must happen instead | |---|---| | `price`, `amount`, `total` | Recompute server-side from the catalog and the cart | | `role`, `isAdmin`, `plan`, `tier` | Read from the server's own record for the authenticated user | | `userId`, `ownerId`, `tenantId` | Derive from the session; never accept from the body | | `status`, `verified`, `approved` | Set only by the server-side state machine | | `createdAt`, `updatedAt` | Server clock | | `discount`, `credits` | Look up and validate server-side | Mass assignment is the mechanism. `Object.assign(user, req.body)` or `Model.update(**request.json)` hands the caller your entire schema. Use explicit allowlists or strict schemas that reject unknown fields. ```ts const UpdateProfile = z.object({ displayName: z.string().min(1).max(80), bio: z.string().max(500).optional(), }).strict(); // unknown keys -> error, not silently ignored ``` ## 2. Runtime validation at every trust boundary TypeScript types are erased at runtime. A handler typed `(req: Request<{}, {}, CreateOrder>)` has exactly zero runtime guarantees, and AI-generated code treats that annotation as if it were enforcement. Validate at the boundary — body, query, params, headers you read, webhook payloads, model output, and anything read from a queue. Use Zod, Valibot, Pydantic, or JSON Schema. Prefer `.strict()` so extra fields fail loudly. The payoff beyond correctness: a validated boundary makes fuzzing productive. Point Schemathesis at your OpenAPI spec and it will find the unvalidated edges in minutes without you writing a test. ## 3. Error handling OWASP added **A10:2025 Mishandling of Exceptional Conditions** for this. Two failure modes: **Leaking internals.** A stack trace in a 500 response gives away file paths, library versions, and schema. Map internal errors to generic client messages carrying a correlation ID; put the detail in the log. **Failing open.** An exception in an authorization path that gets caught by a generic handler and falls through to "allow" is a total bypass. Authorization code must fail closed by construction: ```ts // wrong: any throw inside canAccess() becomes "allowed" try { if (await canAccess(user, doc)) return doc; } catch { /* swallowed */ } return doc; // <- the bug // right let allowed = false; try { allowed = await canAccess(user, doc); } catch (e) { log.error(e); allowed = false; } if (!allowed) throw forbidden(); ``` Return the same status for "not found" and "not authorized" on resources whose existence is itself sensitive — otherwise 403-vs-404 is an enumeration oracle. ## 4. External calls Every outbound call needs four things, and AI-generated code typically has none of them: 1. **A timeout.** Default HTTP clients often have none. A hanging call holds a connection until something else breaks. 2. **Bounded retries with exponential backoff and jitter** — and only for idempotent operations. Retrying a charge is worse than failing it. 3. **A circuit breaker** so repeated failure stops hammering a struggling provider. 4. **A defined degraded behavior.** Fail closed, fail open, queue, or serve stale — decide per dependency, in writing. Put all of it in one wrapper per provider. Scattered SDK calls make caching, cost tracking, retries, and provider swaps impossible. Enforce the boundary with a lint rule: ```json "no-restricted-imports": ["error", { "paths": [{ "name": "stripe", "message": "Use src/services/payments instead." }] }] ``` ## 5. Long work does not belong in a request Serverless platforms have hard timeouts. PDF generation, bulk email, imports, image processing, and multi-step AI chains all exceed them at realistic sizes, and the failure mode is a 504 with half-completed state. Measure p95 per endpoint. Anything approaching the platform limit moves to a queue: accept the request, return an ID, do the work in a worker, expose status via polling or a realtime channel. Jobs need: idempotency, bounded retries, a dead-letter queue, visibility into depth and age, and alerts on failure. **Silent job failure is the most common invisible outage** — nothing returns 500, work just stops happening. Add a dead-man's-switch monitor for crons: the alert fires when a job *does not* run. ## 6. Bounded responses An unbounded list endpoint is a denial-of-service primitive and a convenient exfiltration tool. CWE-770 (allocation without limits) is on the 2025 CWE Top 25. - Enforce a maximum page size server-side; clamp rather than error - Allowlist sortable and filterable fields — passing user input into `ORDER BY` is injection - Cap request body size at the edge - Cap the depth and cost of GraphQL queries, and disable introspection in production ## 7. Contracts For anything with a separate frontend, a mobile client, or an external consumer, an explicit contract is the difference between a change that breaks a client at deploy time and one that breaks it in CI. OpenAPI plus generated client types, or a typed RPC layer (tRPC, ts-rest) where both ends are TypeScript. Then: ```bash uvx schemathesis run https://staging.example.com/openapi.json # property-based fuzzing oasdiff breaking old.yaml new.yaml # breaking-change detection in CI ``` Schemathesis is unusually high-value for AI-built APIs: it requires no test authoring, and it finds precisely the failure profile these codebases have — edge-case inputs producing 500s, responses that drift from the documented shape, and validation that accepts data it should reject. Full consumer-driven contract testing (Pact) is usually overkill below several independently-deployed services. Schema-first plus runtime validation gets most of the value at a fraction of the operational cost. --- # Frontend, UX and Accessibility > Covers checks FE-01..08. Domain D05, weight 6. The demo works because the demo is the happy path on a fast laptop with clean data. Production is slow networks, old phones, Safari, empty states, apostrophes, and people clicking the button twice. --- ## 1. The four states Every component that fetches data has four states, and AI-generated components typically implement one. | State | The failure when it is missing | |---|---| | **Loading** | Blank screen; user assumes it is broken and reloads, doubling the load | | **Error** | Blank screen or a raw error object; no way to recover | | **Empty** | Looks identical to loading; new users conclude the product does not work | | **Success** | The one that exists | Error states must be actionable: what happened in human language, what the user can do, and a retry that does not require a page reload. "Something went wrong" with no retry trains people to leave. Empty states are a product surface, not an edge case. A new user's first screen is almost always the empty state — it should explain what goes here and how to add the first one. ## 2. Error boundaries An unhandled render error white-screens the entire application. Boundaries at route level and around major independent sections turn a total outage into a broken panel. ```tsx This section could not load. } onError={(err, info) => reportError(err, { component: info.componentStack })} > ``` Boundaries must report, not just catch. A silently swallowed error is worse than a crash, because now it is invisible in both directions. Note the limits: React error boundaries do not catch errors in event handlers, async code, or SSR. Those need explicit try/catch and a global handler (`window.onerror`, `unhandledrejection`). ## 3. The real device matrix Test the top three journeys on: - **An actual phone**, not a resized browser window. Touch targets, keyboard overlay behavior, and viewport units differ. - **Safari.** Date parsing, `100vh` behavior, storage restrictions in private mode, and IndexedDB quirks break apps that work everywhere else. - **A throttled network.** Slow 4G in devtools. Race conditions and missing loading states surface immediately. - **An older device** if your audience has one. A three-year-old Android is a different performance class. Record the matrix with pass/fail and notes. That table is the artifact for FE-03. ## 4. Hostile input The reliable crash set, all of which should be tested on every form: - Empty submission - Apostrophes and quotes: `O'Brien`, `"quoted"` - Emoji and non-Latin scripts (a 4-byte emoji breaks anything assuming 1 byte per character) - 10,000 characters pasted into a text field - Leading and trailing whitespace - Double submit — click the button twice fast - Browser back during a submission - Session expiring mid-form - Slow network: submit, then wait Fixes: disable the submit control while in flight, preserve entered data on failure, enforce `maxlength` server-side as well as in the input, and make the form recoverable after an auth expiry rather than dumping the user to a login page with their work gone. ## 5. Accessibility Two numbers set the strategy. The WebAIM Million study of the top million home pages in February 2026 found **95.9% had detectable WCAG failures**, and **six mechanical defects accounted for about 96% of all detected errors**: low-contrast text, missing image alt text, missing form labels, empty links, empty buttons, and missing document language. All six are automatable. Wiring `axe-core` into your test suite eliminates nearly all of them for near-zero ongoing effort: ```ts import AxeBuilder from '@axe-core/playwright'; test('dashboard has no automatically detectable a11y violations', async ({ page }) => { await page.goto('/dashboard'); const results = await new AxeBuilder({ page }) .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']) .analyze(); expect(results.violations).toEqual([]); }); ``` The honest limit: automated tooling covers roughly 20–30% of WCAG success criteria, even though that represents around 57% of issue *instances*. Both numbers are true and they answer different questions. Everything conceptual is invisible to a scanner — focus order, whether alt text is meaningful rather than merely present, heading hierarchy semantics, keyboard traps in custom widgets, and whether a screen reader announcement makes sense. So there is a small mandatory manual layer, roughly 30–60 minutes per release: 1. **Unplug the mouse.** Tab through each critical journey. Visible focus at every step, logical order, no traps, `Esc` closes modals, focus returns to where it came from. 2. **One screen reader pass** on the top two flows — VoiceOver with Safari, or NVDA with Firefox. 3. **200% zoom and a 320px-wide viewport.** Content must reflow without horizontal scrolling. That plus axe in CI puts a small team ahead of most of the market, and it is the difference between a legal exposure and a defensible position. ## 6. Performance Measure on real users, not just in the lab. Lab tools tell you what is possible on a good connection; field data tells you what your users actually get. Track Largest Contentful Paint, Interaction to Next Paint, and Cumulative Layout Shift at the 75th percentile. Verify the current recommended thresholds against Google's published guidance rather than assuming — they have changed, and INP replaced FID. Practical levers, in the order they usually pay: - Ship less JavaScript. Analyze the bundle; the biggest win is usually one accidentally-imported heavy dependency. - Modern image formats, correct dimensions, lazy loading below the fold, and explicit width/height to prevent layout shift. - `font-display: swap` and preloaded fonts. - Code-split by route. - Set a bundle-size budget and fail the build when it is exceeded. Without a gate, bundle size only goes one direction. ## 7. Five users Watch five people from your target audience attempt the core task with no explanation and no help. Say nothing while they do it — this is the hard part. Record: did they complete it, how long, and where they stalled. Three people stalling in the same place is not a coincidence and it is not their fault. This is the cheapest high-value check in the entire catalog and the one most consistently skipped, because the builder can always use their own product and mistakes that for evidence. --- # Application Security > Covers checks APPSEC-01..11. Domain D06, weight 9. The 2025 OWASP Top 10 reordered around what actually happens: Broken Access Control first, Security Misconfiguration second, and a new **Software Supply Chain Failures** category third — voted the number one risk by half of surveyed practitioners. SSRF was folded into Broken Access Control, and a new category was added for **Mishandling of Exceptional Conditions**. Access control lives in `02-auth-and-authorization.md`; supply chain in `08-supply-chain.md`. This file covers the rest. --- ## 1. Injection: structural prevention only **SQL injection** is CWE-89, second on the 2025 CWE Top 25. Parameterize, always, without exception. An ORM handles this until someone reaches for a raw query — grep for those specifically. The dynamic-identifier case is the one that catches people, because parameters cannot bind identifiers: ```ts // Cannot parameterize a column name. Allowlist it. const SORTABLE = new Set(["created_at", "name", "updated_at"]); if (!SORTABLE.has(sort)) throw badRequest(); const rows = await sql`SELECT * FROM docs ORDER BY ${sql(sort)} LIMIT ${limit}`; ``` **Command injection** is CWE-78 at #9. Never pass user input to a shell. Use the array form of process spawning, never string concatenation, and never `shell: true`. If a filename must reach a command, validate it against a strict pattern first. Also in the family: LDAP, XPath, template injection (a user-controlled template string in Jinja or Handlebars is remote code execution), and NoSQL operator injection (`{"$ne": null}` submitted where a string was expected). ## 2. Output encoding — the AI weak spot This deserves its own emphasis because the data is stark. Veracode's longitudinal study of AI-generated code found models pass SQL-injection defense tests around 82% of the time, and **XSS defense tests around 15%**. Models learned to parameterize queries and never learned to encode output. XSS is CWE-79, first on the 2025 CWE Top 25. **If you test one thing in an AI-built codebase, test output encoding.** - Use the framework's automatic escaping and audit every escape hatch: `dangerouslySetInnerHTML`, `v-html`, `innerHTML`, `outerHTML`, `document.write`, `insertAdjacentHTML`, `{{{ }}}`. - Sanitize rich text with DOMPurify — server-side, since client-side sanitization is bypassable. - Encode for the specific context. HTML-escaping a value that lands in a JavaScript string or a URL does not protect it. - Validate URL schemes before rendering user-supplied links — `javascript:` and `data:` are the payload. - Markdown renderers must have raw HTML disabled, or the output must be sanitized. - A strict CSP is your compensating control when encoding fails, which is why the two belong together. ```bash grep -rn "dangerouslySetInnerHTML\|v-html\|innerHTML\s*=\|insertAdjacentHTML\|document.write" \ --include="*.{ts,tsx,js,jsx,vue,svelte}" src/ ``` Every hit needs a justification and a sanitizer. ## 3. Security headers The current baseline, verified against the OWASP Secure Headers project: ``` Strict-Transport-Security: max-age=63072000; includeSubDomains X-Content-Type-Options: nosniff Referrer-Policy: no-referrer Cross-Origin-Opener-Policy: same-origin Cross-Origin-Resource-Policy: same-origin X-Frame-Options: DENY Permissions-Policy: camera=(), microphone=(), geolocation=(), payment=(), usb=(), interest-cohort=() Content-Security-Policy: script-src 'nonce-{RANDOM}' 'strict-dynamic'; object-src 'none'; base-uri 'none'; frame-ancestors 'none' ``` Notes that matter: - **The strict, nonce-based CSP is the current recommendation**, not an allowlist of domains. Allowlist CSPs are routinely bypassable via JSONP endpoints and hosted script CDNs. - The nonce must be **unique per response** and cryptographically random. Do not build middleware that auto-injects the nonce into every ``, `'; DROP TABLE users;--` - Emoji and non-Latin script in every text field - 10,000 characters pasted in - Leading and trailing whitespace - Double-click submit - Browser back mid-request, then forward - Refresh mid-request - Session expires with the form filled in - Slow 4G throttling - Airplane mode, then reconnect - Safari on an actual iPhone - An older Android device - Screen reader on the primary flow - Keyboard only, no mouse - 200% browser zoom - 320px viewport width **Pass condition:** no blank screens, no lost input, no duplicate side effects, no unhandled exceptions in the console. ## 7. Restore drill (DATA-02) 1. Note the current time and the latest available backup timestamp 2. Restore into a **scratch** environment — never over anything real 3. Point a running application at it 4. Verify: row counts on the three largest tables, one specific known record, extensions, enums, sequences, and that the app boots and a user can log in 5. **Record elapsed time.** That is your RTO. 6. Note the gap between backup timestamp and now. That is your RPO. 7. Destroy the scratch environment ## 8. Dependency failure drill (REL-07, API-04) For each critical provider: point the client at a black-hole endpoint (a host that accepts the connection and never responds) and observe. | Check | Pass condition | |---|---| | Does the request time out? | Yes, within the configured timeout | | Does the app stay up? | Yes; other endpoints unaffected | | Does the connection pool survive? | Yes; no cascade | | Does the user see something useful? | Yes; specific message, not a spinner | | Does it alert? | Yes | | Does the retry back off? | Yes; no tight loop | ## 9. Deploy and rollback drill (CI-05) While nothing is on fire: 1. Deploy a trivial visible change 2. Confirm it is live and note the version identifier 3. **Roll back** 4. Confirm the previous version is live 5. Record elapsed time and the exact commands used 6. Note explicitly what the rollback did **not** undo — migrations, sent emails, external side effects, cache state --- # Failure Modes of AI-Generated Code This is the review checklist. When a human reads AI-written code — which check TEST-08 requires for anything security-critical — these are the things to look for, ordered by how often they appear and how much they cost. The evidence behind the ordering is worth knowing, because it changes where you point your attention. --- ## What the research actually shows **Security has not improved with model capability.** A longitudinal study running since 2023 across more than 150 models, using 80 coding tasks in four languages, found the security pass rate flat at around 55% — meaning roughly **45% of generated code contains a vulnerability** — while syntactic correctness rose above 95%. Frontier models cluster at the same security number as their predecessors. Reasoning-focused models reach around 70–72%, better but not production-acceptable. **The per-vulnerability breakdown is the actionable part.** SQL injection defense passes around 82% of the time and insecure cryptography around 86%. **Cross-site scripting passes around 15%, and log injection around 13%.** Models learned to parameterize queries and never learned to encode output. If you audit one thing, audit output encoding. **Defects moved up the stack.** A large enterprise study comparing AI-assisted repositories before and after adoption found syntax errors down 76% and logic bugs down 60%, alongside roughly ten times more security findings — **privilege escalation paths up 322%** and **architectural design flaws up 153%**, with exposed credentials roughly twice as common. The summary line is the most useful one-sentence description of the problem: AI is fixing the typos and creating the timebombs. **Maintainability is degrading measurably.** An analysis of 623 million changed lines from 2023 to 2026 found duplicated code blocks up 81%, copy-paste share rising from 9.4% to 15.7%, and — the striking one — **moved code, the proxy for refactoring, collapsing from 21% to 3.8%**. Cross-file function calls are down 35%, meaning new code increasingly does not call your existing code. It reimplements it. That is how you end up with five slightly different `formatCurrency` functions and a fix that lands in one of them. **Self-assessment is unreliable.** A randomized trial of experienced developers on their own repositories found them 19% *slower* with AI assistance while predicting a 24% speedup — and still believing afterwards they had been sped up by 20%. A separate controlled study found participants with an AI assistant wrote less secure code and were *more* confident it was secure. That last finding is the one that justifies this entire skill. **Verification cannot be delegated to the judgement of the person who wrote it with AI**, because that judgement has been measured and it is wrong in a consistent direction. --- ## The review checklist ### Authorization and access — check first, always - [ ] Handler checks **ownership**, not just authentication - [ ] Tenant filter present in list **and** detail **and** search **and** export queries - [ ] Role read from server state, never from a client-supplied claim - [ ] New endpoint added to the route inventory and to the authorization test matrix - [ ] Nested/related-record access does not traverse past the ownership boundary - [ ] Bulk endpoints apply the same scoping as single-record ones ### Output encoding — the statistically weakest area - [ ] No unsanitized raw-HTML sink (`dangerouslySetInnerHTML`, `v-html`, `innerHTML`) - [ ] User-supplied URLs validated for scheme before rendering - [ ] Markdown renderer has raw HTML disabled or output sanitized - [ ] Values encoded for the context they land in, not just HTML-escaped - [ ] Log statements do not interpolate unescaped user input ### Input and trust boundaries - [ ] Runtime schema validation, not just a TypeScript annotation - [ ] Strict mode — unknown fields rejected, not silently ignored - [ ] No mass assignment (`Object.assign(model, req.body)`) - [ ] Price, role, tier, ownership, and status recomputed server-side - [ ] File uploads validated by content, not extension ### Data integrity - [ ] Migration is a file, not a console command - [ ] Migration linted for locks and destructive operations - [ ] Foreign keys, unique constraints, and `NOT NULL` present where meaningful - [ ] Ownership columns on any table holding user data - [ ] Idempotency for anything with an external side effect - [ ] Transactions wrap multi-step writes; no partial-state windows ### Error handling - [ ] Every `catch` either handles or rethrows — no silent swallow - [ ] Authorization code **fails closed** on exception - [ ] Client responses carry no stack trace, path, or schema detail - [ ] Empty result distinguished from error result - [ ] Promise rejections handled; no unhandled rejection warnings ### External calls - [ ] Explicit timeout - [ ] Retries bounded, with backoff, and only for idempotent operations - [ ] Defined behavior when the provider is down - [ ] Call goes through the provider wrapper, not a direct SDK import ### Dependencies - [ ] Every newly added package **actually exists** and is the one intended - [ ] Package predates the pull request; not created last week with a plausible name - [ ] No unnecessary new dependency for something the standard library does - [ ] Lockfile updated; install scripts not newly enabled ### Secrets - [ ] No credential in the diff, in any form - [ ] No credential behind a client-exposed environment prefix - [ ] No credential in a log line, error message, or test fixture ### Duplication and integration - [ ] Does this reimplement something that already exists in the codebase? - [ ] Does it call existing helpers, or a parallel private copy? - [ ] Are there now two sources of truth for the same rule? ### Agent-specific - [ ] No production credential reachable by an agent - [ ] New tool scoped to its own least-privilege credential - [ ] Irreversible action gated behind approval showing full parameters - [ ] Model output validated before reaching any sink --- ## Specific patterns to grep for ```bash # authorization omissions: findUnique/findById without a scoping clause grep -rn "find\(Unique\|First\|ById\)\?(" --include="*.ts" src/ | grep -v "userId\|tenantId\|ownerId\|accountId" # raw HTML sinks grep -rn "dangerouslySetInnerHTML\|v-html\|innerHTML\s*=\|insertAdjacentHTML\|document.write" src/ # swallowed errors grep -rn "catch\s*(\s*\(.*\)\?\s*)\s*{\s*}" -A1 src/ grep -rn "except.*:\s*pass" --include="*.py" . # string-built queries grep -rn "\(query\|execute\|raw\)(.*\(\`\|+\).*\${\?" src/ # shell execution grep -rn "exec(\|execSync(\|shell\s*[:=]\s*[Tt]rue\|os.system(" src/ # client-exposed env vars grep -rn "NEXT_PUBLIC_\|VITE_\|REACT_APP_\|EXPO_PUBLIC_" src/ | grep -iE "secret|key|token|password" # TODO/FIXME left by the model grep -rn "TODO\|FIXME\|XXX\|HACK\|for now\|temporary" src/ | head -40 ``` That last one is worth running. Models leave honest markers about the shortcuts they took, and nobody reads them. --- ## Process implications Three changes to how work is reviewed, each following directly from the evidence: 1. **Review the wiring, not the syntax.** The compiler and the linter already cover what AI got better at. Human attention belongs on authorization, transactions, error paths, and integration — where the defects moved. 2. **Keep pull requests small.** AI-assisted changes concentrate in fewer, larger pull requests, and large diffs get rubber-stamped. One documented case had a single AI-driven pull request altering authorization headers across multiple services. A 40-file diff does not get reviewed; it gets approved. 3. **Prefer external verification to judgement.** Tests, CI gates, fuzzers, linters, and scanners do not have the confidence-calibration problem that measurement has repeatedly found in humans working with AI. Build the gate rather than resolving to be careful. --- # Command Appendix Copy-paste starting points. **Verify current flags and versions against each tool's documentation** — this appendix will drift, and the surrounding files tell you what to accomplish, not which flag spelling is current this month. Never run a destructive command without a confirmed backup. --- ## Reconnaissance ```bash ls -la; cat README* 2>/dev/null | head -60 cat package.json requirements.txt pyproject.toml go.mod Gemfile 2>/dev/null # env var names only, never values grep -rhoE "(process\.env\.|import\.meta\.env\.|os\.environ)[A-Za-z_\[\"']*[A-Z_][A-Z0-9_]*" . 2>/dev/null \ | grep -oE "[A-Z_][A-Z0-9_]{2,}$" | sort -u # vendors grep -rEl "stripe|paddle|openai|anthropic|gemini|supabase|firebase|clerk|auth0|twilio|sendgrid|resend|s3|cloudinary" \ --include="*.{ts,tsx,js,jsx,py,go,rb}" . 2>/dev/null | head -40 # surface ls .github/workflows/ migrations/ prisma/migrations/ supabase/migrations/ 2>/dev/null git log --oneline | head -20 git log --format='%an' | sort | uniq -c | sort -rn ``` ## Secrets ```bash gitleaks git -v . gitleaks dir --redact --report-format sarif --report-path gitleaks.sarif . trufflehog git file://. --results=verified trufflehog filesystem . --results=verified,unknown git log --all --full-history -- .env .env.local .env.production find . -name "*.env*" -not -path "*/node_modules/*" # client bundle npm run build && grep -rohE "(sk_live_|sk_test_|ghp_|github_pat_|AKIA[0-9A-Z]{16}|AIza[0-9A-Za-z_-]{35}|xox[baprs]-)" \ dist/ build/ .next/static/ .output/ 2>/dev/null | sort -u ``` Pre-commit: ```yaml repos: - repo: https://github.com/gitleaks/gitleaks rev: hooks: [{ id: gitleaks }] ``` ## Dependencies and supply chain ```bash npm ci --ignore-scripts npm audit --audit-level=high --omit=dev npm audit signatures --include-attestations npm outdated trivy fs --scanners vuln,secret,misconfig . osv-scanner scan source -r . grype dir:. syft . -o cyclonedx-json=sbom.cdx.json npm sbom --sbom-format cyclonedx > sbom.cdx.json ``` Cooldown configuration: ```ini # .npmrc — units are days min-release-age=3 ``` ```yaml # pnpm-workspace.yaml — units are minutes minimumReleaseAge: 4320 minimumReleaseAgeStrict: true ``` ## Static and dynamic analysis ```bash semgrep scan --config auto . semgrep scan --config auto --error . # exit non-zero on findings semgrep scan --config auto --baseline-commit=origin/main # diff-scoped docker run -e GITHUB_AUTH_TOKEN=$TOKEN ghcr.io/ossf/scorecard:stable \ --show-details --repo=https://github.com/ORG/REPO docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://staging.example.com docker run -t ghcr.io/zaproxy/zaproxy:stable zap-api-scan.py -t https://staging.example.com/openapi.json -f openapi ``` ## Headers and TLS ```bash curl -sI https://yourapp.com | sed -n '1,40p' curl -sI https://yourapp.com | grep -iE "strict-transport|content-security|x-content-type|referrer|permissions-policy|x-frame" echo | openssl s_client -connect yourapp.com:443 -servername yourapp.com 2>/dev/null | openssl x509 -noout -dates ``` ## Database ```sql CREATE EXTENSION IF NOT EXISTS pg_stat_statements; SELECT calls, round(mean_exec_time::numeric,2) AS avg_ms, round(total_exec_time::numeric) AS total_ms, query FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20; SELECT relname, indexrelname, idx_scan FROM pg_stat_user_indexes WHERE idx_scan = 0 ORDER BY relname; SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 20; -- tables with a tenant column but no RLS SELECT c.relname FROM pg_class c JOIN pg_attribute a ON a.attrelid = c.oid AND a.attname IN ('tenant_id','org_id','account_id') WHERE c.relkind = 'r' AND NOT c.relrowsecurity; SHOW max_connections; SELECT count(*), state FROM pg_stat_activity GROUP BY state; ``` Migration linting: ```bash npx squawk migrations/*.sql atlas migrate lint --dev-url "docker://postgres/16/dev" --latest 10 ``` ## Testing ```bash npx playwright test npx playwright test --grep @smoke npx playwright test --shard=1/4 npx playwright show-trace trace.zip uvx schemathesis run https://staging.example.com/openapi.json npx stryker run --mutate "src/{auth,billing}/**/*.ts" oasdiff breaking old.yaml new.yaml ``` Accessibility in Playwright: ```ts import AxeBuilder from '@axe-core/playwright'; const r = await new AxeBuilder({ page }) .withTags(['wcag2a','wcag2aa','wcag21a','wcag21aa']).analyze(); expect(r.violations).toEqual([]); ``` ## Load testing ```bash k6 run --env BASE_URL=https://staging.example.com load-test.js k6 run --vus 50 --duration 5m load-test.js artillery run artillery.yml ``` ## FINISHER itself ```bash python3 scripts/finisher_score.py init --context finisher/context.json --project "myapp" -o finisher/state.json python3 scripts/finisher_score.py validate python3 scripts/finisher_score.py score python3 scripts/finisher_score.py score --json | jq '.open.P0' python3 scripts/finisher_score.py commit --run-id $(date +%Y-%m-%d)-r1 \ --actions finisher/actions.json --commit-sha $(git rev-parse --short HEAD) python3 scripts/finisher_score.py history ``` --- # Appendix: assets/checks.yaml ```yaml # FINISHER Checks Catalog # Machine-readable source of truth for scoring. Consumed by scripts/finisher_score.py. # # SCALE (per check, 0-4): # 0 ABSENT - not present, or unknown after looking # 1 CLAIMED - exists in some form; no evidence; manual only # 2 IMPLEMENTED- present and works; evidence observed once # 3 VERIFIED - proven by a named artifact (test name, command output, file+line, URL) # 4 ENFORCED - automated so it cannot regress (CI gate, runtime policy, alert) # # TIER: P0 blocks any launch. P1 blocks paid/public launch. P2 blocks scale. # WEIGHT: relative importance inside its domain (1-5). # WHEN: applicability conditions from intake. `always` = every project. # Conditions are ANDed. Prefix `!` to negate. meta: catalog_version: "1.0.0" updated: "2026-07-25" scale: 0: ABSENT 1: CLAIMED 2: IMPLEMENTED 3: VERIFIED 4: ENFORCED # Minimum score required for a check to count as "closed" at each tier. closed_at: P0: 3 P1: 3 P2: 2 # Composite score ceilings while tiers remain open. ceilings: open_P0: 39 open_P1: 69 open_P2: 89 bands: - {min: 0, max: 39, label: "DO NOT LAUNCH"} - {min: 40, max: 59, label: "PRIVATE BETA ONLY"} - {min: 60, max: 74, label: "PAID BETA WITH FIXES"} - {min: 75, max: 89, label: "PRODUCTION READY"} - {min: 90, max: 100, label: "SCALE READY"} # Intake conditions. Set true/false in state.json -> context. conditions: has_users: "App has end users with accounts or sessions" is_public: "Reachable from the public internet" is_multi_tenant: "Data is partitioned by org/tenant/workspace" has_admin: "There are privileged/admin users or internal tools" has_pii: "Collects personal data of any kind" has_payments: "Handles money, subscriptions, or payouts" has_uploads: "Accepts user-uploaded files" has_ugc: "Stores user-generated content shown to others" has_ai: "Calls an LLM or other paid inference API" has_agents: "AI can call tools, browse, or take actions" has_third_party: "Calls paid or rate-limited third-party APIs" has_jobs: "Has background jobs, queues, crons, or webhooks" has_email: "Sends transactional email or SMS" has_mobile: "Ships a native or hybrid app to an app store" eu_users: "Serves users in the EU/EEA/UK" us_state_privacy: "Serves consumers in US states with privacy laws" is_regulated: "HIPAA, PCI, FERPA, GLBA, SOC 2, or similar in scope" has_team: "More than one person will maintain this" domains: - {id: D01, name: "Identity, Auth & Authorization", weight: 12, when: [has_users]} - {id: D02, name: "Secrets & Key Management", weight: 8, when: [always]} - {id: D03, name: "Data Layer, Migrations & Backups", weight: 9, when: [always]} - {id: D04, name: "API & Backend Correctness", weight: 7, when: [always]} - {id: D05, name: "Frontend, UX & Accessibility", weight: 6, when: [always]} - {id: D06, name: "Application Security", weight: 9, when: [always]} - {id: D07, name: "Supply Chain & Dependency Integrity", weight: 6, when: [always]} - {id: D08, name: "CI/CD & Release Engineering", weight: 6, when: [always]} - {id: D09, name: "Environments, Config & Infrastructure", weight: 5, when: [always]} - {id: D10, name: "Observability & Error Tracking", weight: 7, when: [always]} - {id: D11, name: "Reliability, Backup & Incident Response",weight: 7, when: [always]} - {id: D12, name: "Performance, Caching & Scale", weight: 5, when: [always]} - {id: D13, name: "Cost Control & Abuse Prevention", weight: 5, when: [always]} - {id: D14, name: "AI & Agent Safety and Economics", weight: 6, when: [has_ai]} - {id: D15, name: "Privacy, Legal & Compliance", weight: 6, when: [has_pii]} - {id: D16, name: "Testing & Verification", weight: 6, when: [always]} - {id: D17, name: "Mobile & App Store", weight: 6, when: [has_mobile]} - {id: D18, name: "Documentation & Handoff", weight: 4, when: [always]} - {id: D19, name: "Product Truth & Outcome Measurement", weight: 3, when: [always]} checks: # ============================================================ # D01 - IDENTITY, AUTH & AUTHORIZATION # ============================================================ - id: AUTH-01 domain: D01 tier: P0 weight: 5 when: [has_users] title: "Authentication uses a battle-tested provider or framework, not hand-rolled code" risk: "Custom auth is the single most common catastrophic flaw in AI-generated apps. Session fixation, timing attacks, and broken password reset are all default outcomes." fix: "Adopt a managed provider (Auth0, Clerk, WorkOS, Supabase Auth, Cognito, Firebase Auth, Better Auth, NextAuth/Auth.js with a vetted adapter) or your framework's first-party auth. If custom is unavoidable, document why and get it reviewed." evidence: "Name the provider and the file where the session is established. If custom: link the review." enforce: "Lint or CODEOWNERS rule blocking edits to auth files without a named reviewer." - id: AUTH-02 domain: D01 tier: P0 weight: 5 when: [has_users] title: "Every non-public route and API endpoint enforces authentication server-side" risk: "AI scaffolds frequently protect the UI route but leave the underlying API open. The page redirects; the JSON endpoint does not." fix: "Default-deny middleware. Enumerate every route; whitelist public ones explicitly. Never rely on the client hiding a link." evidence: "A route inventory table (path, method, auth required, authz rule) plus a test hitting each protected endpoint unauthenticated and asserting 401/403." enforce: "Automated route-inventory test that fails CI when a new route lacks an explicit auth declaration." - id: AUTHZ-01 domain: D01 tier: P0 weight: 5 when: [has_users] title: "Every resource access checks ownership, not just authentication" risk: "Broken Access Control is OWASP A01:2025 and CWE-862 is #4 on the 2025 CWE Top 25. Authentication proves who you are; it does not stop you loading /invoice/1234." fix: "Every read/update/delete/export filters by the acting user's id, tenant id, or an explicit permission grant. Push the filter into the data layer, not the controller." evidence: "The User A / User B test from references/20-manual-test-scripts.md, executed, with output. IDs enumerated in URL, body, query, and headers." enforce: "Automated IDOR test suite covering every object type, run in CI." - id: AUTHZ-02 domain: D01 tier: P0 weight: 4 when: [is_multi_tenant] title: "Tenant isolation is enforced at the database layer, not only in application code" risk: "One forgotten WHERE clause leaks an entire customer's data. Application-layer-only isolation has no backstop." fix: "Postgres RLS policies, a per-tenant schema/database, or a query-builder wrapper that makes an unscoped query impossible to express. Test with a session set to tenant A querying tenant B rows." evidence: "Policy definitions plus a test that sets the tenant context to A and asserts zero rows returned for B's data." enforce: "CI test asserting RLS is enabled on every table carrying a tenant column; migration lint fails if a new tenant table lacks a policy." - id: AUTHZ-03 domain: D01 tier: P0 weight: 4 when: [has_admin] title: "Admin capability is server-verified and cannot be obtained by client-side manipulation" risk: "isAdmin in localStorage, a role claim the client can edit, or an /admin route protected only by not being linked." fix: "Role lives server-side and is re-checked on every privileged operation. Admin routes are a separate, separately-authorized surface. Consider a distinct hostname and IP or SSO restriction." evidence: "Test: authenticate as a normal user, forge the role client-side, call every admin endpoint, assert 403." enforce: "Admin endpoints share a middleware with its own test suite; CI fails if an admin route bypasses it." - id: AUTHZ-04 domain: D01 tier: P1 weight: 3 when: [has_users] title: "Authorization model is written down and matches the code" risk: "Undocumented permission logic drifts, and nobody can answer 'can a viewer export?' without reading source." fix: "A single table of roles x resources x actions. Generate it from code if possible." evidence: "references/templates/ARCHITECTURE.md permission matrix, cross-checked against the enforcement points." enforce: "Permission matrix is generated from the policy definitions in a CI step." - id: AUTH-03 domain: D01 tier: P0 weight: 4 when: [has_users] title: "Sessions expire, logout truly invalidates, and session IDs rotate on privilege change" risk: "OWASP requires regenerating the session identifier after any privilege change; failing to do so enables session fixation. Client-side-only logout leaves a valid token." fix: "Server-side session invalidation on logout. Idle and absolute timeouts enforced server-side. Regenerate the session ID immediately after login and after any role change." evidence: "Capture a session token, log out, replay the token, assert rejection. Confirm the ID differs pre/post login." enforce: "Integration test in CI covering logout invalidation and rotation." - id: AUTH-04 domain: D01 tier: P1 weight: 3 when: [has_users] title: "Session cookies use Secure, HttpOnly, SameSite, and the __Host- prefix where possible" risk: "A cookie readable by JavaScript turns any XSS into full account takeover." fix: "Set Secure; HttpOnly; SameSite=Lax or Strict. Use the __Host- prefix (requires Secure, Path=/, no Domain attribute) for maximum protection." evidence: "Raw Set-Cookie header from a real login response." enforce: "Test asserting cookie attributes on the login response." - id: AUTH-05 domain: D01 tier: P0 weight: 4 when: [has_users] title: "Password reset and email verification tokens are single-use, short-lived, and unguessable" risk: "A reusable or long-lived reset link is a permanent backdoor. AI-generated reset flows frequently omit invalidation after use." fix: "Cryptographically random token, hashed at rest, expiring in 15-60 minutes, invalidated on use and on password change. Invalidate all sessions on password change." evidence: "Test: use a reset link twice, assert second attempt fails. Test expiry." enforce: "Automated test for reuse, expiry, and session invalidation on reset." - id: AUTH-06 domain: D01 tier: P1 weight: 3 when: [has_users] title: "Login, reset, signup, and OTP endpoints are rate limited per account and per IP" risk: "Credential stuffing and password spraying are automated and cheap. OWASP advises locking on account identity, not IP alone." fix: "Per-identity throttling with exponential backoff plus per-IP limits, tuned against DoS abuse. Generic error text that does not disclose account existence." evidence: "Script 200 failed logins; show the lockout or throttle response." enforce: "Rate limit is configured at the edge/gateway and covered by a test." - id: AUTH-07 domain: D01 tier: P1 weight: 2 when: [has_users] title: "Password policy follows current guidance: length over composition, breach-list blocking" risk: "Composition rules produce predictable passwords and no security. Reused breached passwords are the actual attack." fix: "Minimum 8 with MFA or 15 without; maximum at least 64; allow spaces and Unicode; no forced rotation; check against a breached-password corpus (e.g. Pwned Passwords k-anonymity API)." evidence: "Policy code plus a test rejecting a known-breached password." enforce: "Test in CI." - id: AUTH-08 domain: D01 tier: P1 weight: 2 when: [has_users, has_admin] title: "MFA or passkeys available for users, mandatory for admins" risk: "A single phished admin password is total compromise. The 2025 npm chalk/debug compromise began with one phished maintainer." fix: "Offer TOTP/passkeys (WebAuthn). Require a second factor for all admin and billing-privileged accounts." evidence: "Screenshot or config showing enforcement on an admin account." enforce: "Policy enforced by the identity provider, not optional in the UI." - id: AUTH-09 domain: D01 tier: P2 weight: 2 when: [has_users] title: "OAuth/OIDC integrations use PKCE, exact redirect URI matching, and state/nonce binding" risk: "Pattern-matched redirect URIs and missing PKCE enable authorization code interception. OAuth 2.1 makes PKCE mandatory for all authorization-code clients." fix: "PKCE on every authorization code flow. Exact-string redirect URI registration. Bind state to the user agent; use nonce for OIDC." evidence: "Capture the authorization request; show code_challenge and exact redirect registration." enforce: "Integration test or provider configuration screenshot in the handoff docs." - id: AUTH-10 domain: D01 tier: P1 weight: 3 when: [has_users] title: "Tokens are validated with a pinned algorithm and are revocable" risk: "alg:none and algorithm-confusion attacks remain live because libraries accept whatever the token declares. JWTs have no native revocation." fix: "Pin the expected algorithm at verification. Short access-token TTL plus refresh rotation, or server-side session lookup. Maintain a revocation list keyed by token digest if you must use long-lived tokens." evidence: "Verification code showing the pinned algorithm; a test forging alg:none and asserting rejection." enforce: "Test in CI." - id: AUTH-11 domain: D01 tier: P1 weight: 2 when: [has_users] title: "Account deletion and data export exist and actually work end to end" risk: "Required by GDPR, CCPA, and Apple App Store guideline 5.1.1 for any app that supports account creation." fix: "In-product deletion that removes or irreversibly anonymizes across primary store, caches, search indexes, analytics, and downstream processors. Documented backup expiry window." evidence: "Delete a test account; show the rows/objects gone from every store; document the backup lag." enforce: "Scheduled job verifies deletion propagation; test covers the happy path." - id: AUTH-12 domain: D01 tier: P2 weight: 2 when: [has_admin] title: "Privileged actions are recorded in an append-only audit log" risk: "Without an audit trail you cannot answer 'who deleted it' or satisfy an enterprise security review." fix: "Log actor, action, target, before/after, timestamp, request id, and source IP for every admin and destructive action. Store separately from application logs, write-once." evidence: "Sample audit entries for an admin action." enforce: "Audit write is inside the transaction for privileged mutations; missing-audit test fails CI." # ============================================================ # D02 - SECRETS & KEY MANAGEMENT # ============================================================ - id: SEC-01 domain: D02 tier: P0 weight: 5 when: [always] title: "No secret is reachable from browser or mobile client code" risk: "Anything shipped to a client is public. NEXT_PUBLIC_/VITE_/REACT_APP_ prefixed variables are compiled into the bundle." fix: "Move every privileged call behind a server route. Only publishable/anon keys designed for client exposure may ship." evidence: "Grep the built bundle for sk_, key, secret, token, and each provider's key prefix. Paste the (empty) result. Inspect DevTools Network for outbound calls carrying credentials." enforce: "CI step that greps the production build for secret patterns and fails on a hit." - id: SEC-02 domain: D02 tier: P0 weight: 5 when: [always] title: "Secret scan of the full git history passes, and anything ever exposed has been rotated" risk: "git rm does not remove a secret from history. A leaked key is compromised the moment it is pushed, not when it is discovered." fix: "Run gitleaks and/or trufflehog over all history. Rotate every hit at the provider. Rewriting history is optional; rotating is not." evidence: "`gitleaks git -v .` output plus a rotation log listing key name, date rotated, and who did it." enforce: "gitleaks in pre-commit AND in CI AND server-side push protection enabled." - id: SEC-03 domain: D02 tier: P1 weight: 4 when: [always] title: "Secrets live in a secret manager or platform env store, scoped per environment" risk: "A .env file shared over Slack becomes a permanent, unrotatable, unauditable credential." fix: "Platform env vars or a secret manager (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, Doppler, Infisical). Distinct values per environment. .env.example lists names only." evidence: "Show the store and the per-environment scoping; confirm .env is gitignored and .env.example has no values." enforce: "Deploy fails if a required secret name is missing; no plaintext secrets in the repo, verified by CI." - id: SEC-04 domain: D02 tier: P1 weight: 3 when: [always] title: "CI/CD uses short-lived federated credentials (OIDC), not long-lived cloud keys" risk: "A static AWS key in CI is a permanent credential harvested by every npm supply-chain worm of the last two years." fix: "GitHub Actions OIDC to AWS/GCP/Azure with a role trust policy scoped to repo, branch, and environment. Set permissions: id-token: write only on the jobs that need it." evidence: "Workflow file showing the OIDC exchange and the scoped trust policy subject claim." enforce: "No long-lived cloud keys exist in repository or organization secrets." - id: SEC-05 domain: D02 tier: P1 weight: 3 when: [always] title: "Service accounts and database credentials follow least privilege" risk: "One app-wide superuser credential turns any injection or SSRF into total control." fix: "Separate read/write roles. Migration credentials distinct from runtime credentials. Third-party API keys scoped to the minimum permission set and, where supported, IP-restricted." evidence: "Grant listing per role; note what each cannot do." enforce: "Infrastructure-as-code defines the grants; drift detection or periodic review scheduled." - id: SEC-06 domain: D02 tier: P1 weight: 3 when: [always] title: "Secrets never appear in logs, error messages, traces, screenshots, or AI prompts" risk: "Redaction failures move a secret from a vault into a log aggregator with far broader access." fix: "Redaction middleware on the logger with a deny-list of key names and value patterns. Scrub request bodies and headers. Never paste .env contents into an AI chat." evidence: "Trigger an error carrying a secret-bearing payload; show the redacted log line." enforce: "Unit test on the redactor; log-sample scan in CI or a scheduled job." - id: SEC-07 domain: D02 tier: P2 weight: 2 when: [always] title: "Key rotation is documented, rehearsed, and possible without downtime" risk: "If rotation requires downtime you will not do it during an incident, which is exactly when you must." fix: "Support two valid keys simultaneously during rollover. Write the rotation runbook. Rehearse once." evidence: "Rotation runbook plus a dated record of a rehearsal." enforce: "Rotation is automated or calendared with an owner." - id: SEC-08 domain: D02 tier: P1 weight: 3 when: [has_payments] title: "Webhook endpoints verify provider signatures and reject replays" risk: "An unverified webhook endpoint lets anyone mark invoices paid or grant subscriptions. This is a top-tier P0 in payment systems." fix: "Verify the signature with the provider's library using the raw request body. Enforce a timestamp tolerance. Store processed event IDs for idempotency." evidence: "Send a forged webhook; show rejection. Send the same valid event twice; show one side effect." enforce: "Both tests run in CI." # ============================================================ # D03 - DATA LAYER, MIGRATIONS & BACKUPS # ============================================================ - id: DATA-01 domain: D03 tier: P0 weight: 5 when: [always] title: "Automated backups exist for every production datastore" risk: "Without backups, one bad migration or one deletion ends the business. AI agents have deleted production databases." fix: "Enable managed automated backups plus point-in-time recovery where available. Include object storage, not just the primary database." evidence: "Backup configuration screenshot/CLI output showing schedule and retention, for each datastore." enforce: "Backup failure raises an alert; monitored, not assumed." - id: DATA-02 domain: D03 tier: P0 weight: 5 when: [always] title: "A restore has actually been performed and the result verified" risk: "An untested backup is a hypothesis. Silent corruption, missing extensions, and wrong retention windows only surface on restore." fix: "Restore to a scratch environment. Run the app against it. Verify row counts and a known record. Record elapsed time as your real RTO." evidence: "Dated restore log with elapsed time and verification queries." enforce: "Restore drill scheduled at a defined cadence with a named owner." - id: DATA-03 domain: D03 tier: P0 weight: 4 when: [always] title: "Production, staging, and development use separate datastores" risk: "Shared databases mean a dev seed script can wipe customer data. This is the root cause of the best-known AI-agent data-loss incident." fix: "Separate instances or database branches. Different credentials. Production credentials are not present in any developer environment." evidence: "Connection string hostnames per environment (redacted) showing they differ." enforce: "Startup assertion that refuses to run destructive scripts against a production host; production credentials absent from dev tooling." - id: DATA-04 domain: D03 tier: P0 weight: 4 when: [always] title: "No human or agent has standing write access to the production database" risk: "Direct production edits are unlogged, unrepeatable, and undo your migrations. Agents with production credentials is the highest-severity configuration in this entire catalog." fix: "Read-only by default. Break-glass write access requires an explicit, logged, time-boxed elevation. AI coding tools get development credentials only." evidence: "Access listing showing who has write, and the elevation procedure." enforce: "Break-glass access is time-boxed and alerts on use." - id: DATA-05 domain: D03 tier: P1 weight: 4 when: [always] title: "Every schema change is a versioned migration file in the repo" risk: "Schema drift makes environments non-reproducible and rollback impossible." fix: "Use the framework's migration tool. No console DDL. A fresh database must be creatable from migrations alone." evidence: "Create an empty database, run migrations, boot the app. Paste the output." enforce: "CI job that migrates a clean database from zero on every PR." - id: DATA-06 domain: D03 tier: P1 weight: 4 when: [always] title: "Migrations are linted for locking and destructive operations" risk: "An AI-written migration that adds a unique constraint or a non-concurrent index takes an exclusive lock and halts your app under load. Destructive column drops lose data silently." fix: "Run squawk (Postgres) or Atlas lint as a required PR check. Adopt expand/contract: additive change, dual-write, backfill, switch reads, then contract." evidence: "Lint output on the current migration set." enforce: "Migration lint is a required status check on PRs." - id: DATA-07 domain: D03 tier: P1 weight: 3 when: [always] title: "Rollback strategy for schema changes is defined and it is roll-forward-safe" risk: "Down migrations are rarely exercised and frequently lose data on the worst possible day." fix: "Prefer backwards-compatible migrations so a code rollback is sufficient. Test that version N-1 of the app still works against version N of the schema." evidence: "Test or documented procedure showing the previous release runs against the new schema." enforce: "CI runs the previous release's test suite against the migrated schema." - id: DATA-08 domain: D03 tier: P1 weight: 3 when: [always] title: "Constraints and indexes reflect real access patterns" risk: "AI-generated schemas routinely omit foreign keys, unique constraints, and NOT NULL, pushing integrity enforcement into application code that forgets it." fix: "Foreign keys with deliberate cascade rules, unique constraints on natural keys, NOT NULL where truly required, check constraints on enums. Index from measured slow queries, not guesses." evidence: "Schema dump plus the slow-query report the indexes were derived from." enforce: "pg_stat_statements or equivalent reviewed at a defined cadence; slow-query alert configured." - id: DATA-09 domain: D03 tier: P1 weight: 3 when: [has_uploads] title: "File uploads enforce size, type, and content validation, and are stored outside the app server" risk: "Unrestricted file upload is CWE-434, #12 on the 2025 CWE Top 25. Extension checks are trivially bypassed." fix: "Server-side size cap, content-type sniffing against an allowlist, randomized stored filenames, storage in object storage with private ACLs and signed URLs, never executing uploaded content from a web-served path." evidence: "Attempt to upload an oversized file, a disguised executable, and an SVG containing script; show all three rejected or neutralized." enforce: "Upload validation tests in CI; bucket policy prevents public listing." - id: DATA-10 domain: D03 tier: P1 weight: 3 when: [has_pii] title: "Data retention and deletion rules are defined per data class" risk: "Keeping everything forever maximizes breach impact and violates data-minimization principles." fix: "Classify each table/bucket: what it holds, why, how long, who can read it. Implement scheduled purges. Define backup expiry so deletion eventually propagates." evidence: "Data inventory table with retention periods and the job that enforces them." enforce: "Purge job runs on schedule and is monitored." - id: DATA-11 domain: D03 tier: P2 weight: 2 when: [has_pii] title: "Sensitive fields are encrypted at rest beyond disk-level encryption where warranted" risk: "Disk encryption protects against stolen hardware, not against a leaked database credential." fix: "Application-level or column-level encryption for the highest-sensitivity fields (government IDs, health data, financial account numbers). Use a managed KMS." evidence: "Show the encrypted column and the key management path." enforce: "Encryption applied at the ORM/model layer so new writes cannot bypass it." - id: DATA-12 domain: D03 tier: P1 weight: 3 when: [always] title: "Idempotency keys prevent duplicate side effects from retries and double-clicks" risk: "Double-charging, duplicate orders, and duplicate emails are the classic symptom of a missing idempotency layer. AI-generated handlers almost never include one." fix: "Client-supplied or server-derived idempotency key stored with a uniqueness constraint; return the prior result on replay. Apply to payments, sends, and any externally visible side effect." evidence: "Fire the same request twice concurrently; show one side effect." enforce: "Concurrency test in CI." # ============================================================ # D04 - API & BACKEND CORRECTNESS # ============================================================ - id: API-01 domain: D04 tier: P0 weight: 5 when: [always] title: "All privileged operations execute server-side; the client is never trusted" risk: "Business rules enforced only in the UI are advisory. Price, quantity, role, and entitlement must be recomputed server-side." fix: "Recompute prices and entitlements from server state. Never accept a client-supplied amount, role, tier, or ownership field." evidence: "Tamper with a request body (price, role, userId) via curl; show rejection or server-side override." enforce: "Tests for each mass-assignment-sensitive endpoint." - id: API-02 domain: D04 tier: P1 weight: 4 when: [always] title: "Every input is validated against a schema at the trust boundary, at runtime" risk: "TypeScript types are erased at runtime. AI handlers routinely treat req.body as if the declared type were enforced." fix: "Zod/Valibot/Pydantic/JSON Schema validation on every request body, query, param, and header you read. Reject unknown fields (strict mode) to block mass assignment." evidence: "Schema definitions plus a test posting malformed and extra fields, asserting 400." enforce: "Framework-level validation so an unvalidated handler cannot be added; or a lint rule." - id: API-03 domain: D04 tier: P1 weight: 3 when: [always] title: "Errors return safe messages; stack traces and internals never reach clients" risk: "OWASP added A10:2025 Mishandling of Exceptional Conditions specifically for this. CWE-209 leaks schema, paths, and library versions." fix: "Central error handler mapping internal errors to generic client messages plus a correlation ID. Full detail goes to logs only. Never fail open on an exception in an authorization path." evidence: "Force a 500; show the client response contains no stack trace and does contain a correlation ID." enforce: "Test asserting error shape; production config disables debug output." - id: API-04 domain: D04 tier: P1 weight: 3 when: [has_third_party] title: "Every external call has a timeout, bounded retries with backoff and jitter, and a defined fallback" risk: "A hanging third-party call exhausts your connection pool and takes down endpoints that have nothing to do with it." fix: "Explicit timeout on every client. Retry only idempotent operations, with exponential backoff and jitter and a cap. Circuit breaker for repeated failure. Define the degraded behavior." evidence: "Point a client at a black-hole endpoint; show it times out cleanly and the app stays up." enforce: "Timeouts set in a shared HTTP client wrapper, not per call site." - id: API-05 domain: D04 tier: P1 weight: 3 when: [has_third_party] title: "Each external provider is wrapped in a single service module" risk: "Provider calls scattered across 30 files make it impossible to add caching, retries, cost tracking, or to swap providers." fix: "One module per provider exposing domain-shaped functions. All logging, retries, and cost accounting live there." evidence: "Grep for direct SDK usage outside the wrapper; result should be empty." enforce: "Lint rule (e.g. no-restricted-imports) blocking direct SDK imports outside the wrapper." - id: API-06 domain: D04 tier: P1 weight: 3 when: [has_jobs] title: "Work that can exceed the request timeout runs in a background job, not in the request" risk: "PDF generation, bulk email, imports, and multi-step AI chains routinely exceed serverless timeouts, producing 504s and half-completed state." fix: "Queue the work; return an accepted response with a status endpoint or a real-time channel. Jobs must be idempotent and retryable with a dead-letter queue." evidence: "List every operation exceeding 5 seconds at p95 and where it runs." enforce: "Alert on job failures and dead-letter depth." - id: API-07 domain: D04 tier: P1 weight: 3 when: [has_jobs] title: "Background jobs, crons, and webhooks are observable and alert on failure" risk: "Silent job failure is the most common invisible outage: nothing 500s, work just stops happening." fix: "Emit metrics on success, failure, latency, and queue depth. Dead-man's-switch monitoring for crons. Retry with backoff, then dead-letter." evidence: "Show a failed job producing an alert." enforce: "Alert rules configured and tested." - id: API-08 domain: D04 tier: P2 weight: 2 when: [is_public] title: "An API contract exists (OpenAPI or typed RPC) and responses conform to it" risk: "Response drift breaks clients silently and makes handoff and integration testing impossible." fix: "OpenAPI spec, tRPC, or equivalent as the single source of truth. Generate client types from it. Detect breaking changes in CI with oasdiff or similar." evidence: "Spec file plus a conformance run (e.g. schemathesis) showing responses match." enforce: "Breaking-change detection is a required check." - id: API-09 domain: D04 tier: P2 weight: 2 when: [is_public] title: "Pagination, sorting, and filtering are bounded" risk: "An unbounded list endpoint is a denial-of-service primitive and a data-exfiltration convenience. CWE-770 is #25 on the 2025 CWE Top 25." fix: "Enforce a maximum page size server-side. Allowlist sortable and filterable fields to prevent injection via ORDER BY." evidence: "Request limit=1000000; show it clamped." enforce: "Default and max page size set in shared middleware." # ============================================================ # D05 - FRONTEND, UX & ACCESSIBILITY # ============================================================ - id: FE-01 domain: D05 tier: P1 weight: 4 when: [always] title: "Every async surface has loading, error, empty, and success states" risk: "The most common vibe-coded failure: the happy path is beautiful and everything else is a blank screen or an infinite spinner." fix: "Enumerate components fetching data; implement all four states. Errors must be actionable and offer a retry." evidence: "Screenshots or tests for the four states on the top five surfaces." enforce: "Storybook stories or component tests covering each state." - id: FE-02 domain: D05 tier: P1 weight: 3 when: [always] title: "Error boundaries prevent one component failure from blanking the app" risk: "An unhandled render error in a sidebar widget white-screens the entire product." fix: "Error boundaries at route and major-section level, reporting to error tracking, with a recoverable fallback UI." evidence: "Throw deliberately in a child component; show the rest of the app survives and the error was reported." enforce: "Test that asserts the boundary catches and reports." - id: FE-03 domain: D05 tier: P1 weight: 3 when: [always] title: "Core flows verified on real mobile devices, Safari, and a throttled network" risk: "Desktop Chrome on fast wifi is not the deployment environment. Safari-specific date, scroll, and storage behavior breaks apps routinely." fix: "Test the top three journeys on an actual phone, in Safari, and under a 3G/Slow-4G throttle profile. Include an older device if your audience has one." evidence: "Device/browser/network matrix with pass/fail and notes." enforce: "Cross-browser project matrix in the E2E suite." - id: FE-04 domain: D05 tier: P1 weight: 3 when: [always] title: "Forms survive hostile and messy input" risk: "Apostrophes, emoji, 10k-character pastes, double submits, and back-button-mid-request are the reliable crash set." fix: "Test each. Disable submit while in flight. Preserve input on failure. Enforce maxlength server-side too." evidence: "Executed edge-case UX checklist from references/20-manual-test-scripts.md." enforce: "E2E tests covering double-submit and oversized input." - id: FE-05 domain: D05 tier: P1 weight: 3 when: [is_public] title: "Automated accessibility scan passes on core pages" risk: "The WebAIM Million 2026 study found 95.9% of home pages had detected WCAG failures, and six mechanical defects account for ~96% of all errors. These are free to fix." fix: "Wire axe-core into the E2E suite scoped to wcag2a/wcag2aa/wcag21aa. Fix contrast, missing alt text, missing form labels, empty links, empty buttons, and missing document language first." evidence: "axe run output on core pages showing zero violations, or a documented exception list." enforce: "axe assertion in CI on the critical page set." - id: FE-06 domain: D05 tier: P1 weight: 3 when: [is_public] title: "Keyboard-only and screen-reader passes completed on critical journeys" risk: "Automated tooling catches roughly 20-30% of WCAG success criteria. Focus order, keyboard traps, and meaningless labels are invisible to it." fix: "Unplug the mouse and complete each critical journey. Confirm visible focus, logical order, Esc closes modals, focus returns sensibly. One pass with VoiceOver or NVDA on the top flows. Check 200% zoom and 320px reflow." evidence: "Dated notes per journey listing what was found and fixed." enforce: "Repeat each release; recorded in the release checklist." - id: FE-07 domain: D05 tier: P2 weight: 2 when: [is_public] title: "Core Web Vitals measured on real users and within thresholds" risk: "Lab scores mislead. Field data is what search ranking and actual experience reflect." fix: "Collect RUM for LCP, INP, and CLS. Target good thresholds at p75. Set a performance budget for bundle size and block regressions." evidence: "Field data report at p75 for the three metrics." enforce: "Bundle-size budget enforced in CI; RUM dashboard monitored." - id: FE-08 domain: D05 tier: P2 weight: 2 when: [always] title: "Five target users complete the core flow unaided" risk: "The builder can always use their own app. That tells you nothing." fix: "Watch five people from the target audience attempt the core task with no explanation. Record where they stall." evidence: "Notes per participant: completed yes/no, time, stumbling points." enforce: "Repeat after major UX changes." # ============================================================ # D06 - APPLICATION SECURITY # ============================================================ - id: APPSEC-01 domain: D06 tier: P0 weight: 4 when: [always] title: "Injection is prevented structurally, not by escaping strings" risk: "SQL injection is CWE-89, #2 on the 2025 CWE Top 25. Command injection is #9. AI models parameterize SQL reasonably well and handle command construction badly." fix: "Parameterized queries or an ORM everywhere; never build SQL by concatenation. Never pass user input to a shell. Allowlist for any dynamic identifier (table, column, sort field)." evidence: "Grep for string-built queries and shell invocations; show the results are safe. SAST run clean on injection rules." enforce: "Semgrep or CodeQL injection rules in CI, blocking on new findings." - id: APPSEC-02 domain: D06 tier: P0 weight: 4 when: [has_ugc] title: "Output encoding prevents XSS in every rendering context" risk: "Veracode's 2026 longitudinal study found AI models pass XSS defense tests only ~15% of the time, versus 82% for SQL injection. Models learned to parameterize queries and never learned to encode output. XSS is CWE-79, #1 on the 2025 CWE Top 25." fix: "Use framework auto-escaping. Audit every dangerouslySetInnerHTML / v-html / innerHTML. Sanitize rich text with DOMPurify server-side. Context-appropriate encoding for HTML, attribute, URL, JS, and CSS contexts. Never render user-controlled URLs without scheme validation." evidence: "Inventory of raw-HTML sinks with the sanitizer applied at each; a stored-XSS attempt showing neutralization." enforce: "Lint rule flagging raw HTML sinks; SAST XSS rules blocking in CI." - id: APPSEC-03 domain: D06 tier: P1 weight: 4 when: [is_public] title: "Security headers are set, including a strict Content-Security-Policy" risk: "Missing headers turn a small injection into a full compromise and leak referrer data." fix: "Strict-Transport-Security with a long max-age and includeSubDomains; X-Content-Type-Options: nosniff; Referrer-Policy; frame-ancestors plus X-Frame-Options for compatibility; a nonce-based strict CSP with 'strict-dynamic', object-src 'none', base-uri 'none'; a restrictive Permissions-Policy. Do not use X-XSS-Protection (deprecated and itself a risk)." evidence: "curl -I output of production headers, plus a CSP report-only period showing no legitimate violations before enforcing." enforce: "Headers set in middleware/edge config with a test asserting their presence." - id: APPSEC-04 domain: D06 tier: P1 weight: 3 when: [is_public] title: "CORS is restricted to known origins and credentials are not exposed to wildcards" risk: "Access-Control-Allow-Origin reflecting the request origin with credentials enabled is a full same-origin bypass." fix: "Explicit origin allowlist. Never reflect arbitrary origins with credentials. Restrict methods and headers." evidence: "Send a request from an unlisted origin; show the browser blocks it." enforce: "Config-driven allowlist with a test." - id: APPSEC-05 domain: D06 tier: P1 weight: 3 when: [has_users] title: "CSRF protection is present on state-changing requests" risk: "CSRF is CWE-352, #3 on the 2025 CWE Top 25 — unusually high. Cookie-based sessions without SameSite=Strict or tokens are exposed." fix: "SameSite cookies plus anti-CSRF tokens for cookie-authenticated state changes, or a bearer-token scheme not carried automatically by the browser. Verify Origin/Referer on sensitive endpoints." evidence: "Cross-origin state-changing request from a test page; show rejection." enforce: "Framework CSRF middleware enabled globally with an explicit opt-out list." - id: APPSEC-06 domain: D06 tier: P1 weight: 3 when: [always] title: "SAST runs on pull requests and blocks new high/critical findings" risk: "Apiiro measured a 322% increase in privilege-escalation paths and 153% increase in architectural flaws in AI-assisted repositories. Volume defeats manual review." fix: "Semgrep (Community Edition is free) or CodeQL, scoped to the diff on PRs. Baseline existing findings; block only on new high/critical to avoid alert fatigue." evidence: "CI run output plus the baseline file." enforce: "Required status check." - id: APPSEC-07 domain: D06 tier: P1 weight: 3 when: [is_public] title: "SSRF is prevented anywhere the server fetches a user-supplied URL" risk: "SSRF was merged into A01 Broken Access Control in OWASP 2025 and remains CWE-918 at #22. Cloud metadata endpoints turn SSRF into credential theft — this was a named vector in the Shai-Hulud 2.0 worm." fix: "Allowlist destination hosts. Resolve DNS and reject private, loopback, link-local, and metadata ranges — re-check after redirects to defeat DNS rebinding. Disable redirects or re-validate each hop. Block the cloud metadata IP at the network level." evidence: "Attempt fetch of the metadata address and a private IP; show rejection." enforce: "Centralized URL-fetch helper; lint blocking direct HTTP clients for user-supplied URLs." - id: APPSEC-08 domain: D06 tier: P1 weight: 3 when: [is_public] title: "A DAST baseline scan has been run against staging and findings triaged" risk: "Static analysis cannot see misconfiguration, missing headers, or exposed endpoints in the running system." fix: "Run a ZAP baseline (passive) scan against staging. Triage every alert as fix, accept-with-reason, or false positive." evidence: "Scan report plus the triage table." enforce: "Scheduled nightly scan against staging, non-blocking, reviewed." - id: APPSEC-09 domain: D06 tier: P1 weight: 3 when: [always] title: "Manual authorization tampering has been attempted and failed" risk: "No scanner reliably finds broken object-level authorization. It requires a human trying it." fix: "Execute the full auth boundary script: ID substitution in URL, body, query, and headers; role escalation; admin endpoint access; expired session replay; cross-tenant access." evidence: "Completed test log from references/20-manual-test-scripts.md with each attempt and its result." enforce: "Automated IDOR regression suite derived from the manual findings." - id: APPSEC-10 domain: D06 tier: P2 weight: 2 when: [always] title: "A written threat model exists for the highest-value assets" risk: "Without one, controls are chosen by habit rather than by what an attacker would actually do." fix: "For each key asset: who wants it, how they would reach it, what stops them, what detects them. One page is enough. Use references/templates/THREAT-MODEL.md." evidence: "Completed threat model." enforce: "Reviewed when architecture changes materially." - id: APPSEC-11 domain: D06 tier: P2 weight: 2 when: [is_public] title: "A vulnerability disclosure path exists" risk: "Researchers who cannot reach you post publicly instead." fix: "security.txt at /.well-known/security.txt and a monitored security@ address with a stated response commitment." evidence: "Live security.txt URL." enforce: "Inbox monitored with an owner." # ============================================================ # D07 - SUPPLY CHAIN & DEPENDENCY INTEGRITY # ============================================================ - id: SUP-01 domain: D07 tier: P0 weight: 4 when: [always] title: "Install scripts do not run for arbitrary transitive dependencies" risk: "Every major 2025-2026 npm worm executed at install time via preinstall/postinstall, before any application code imported the package. This single control defeats most of them." fix: "On npm 12+, install scripts are off by default; approve only the packages that genuinely need to build native code and commit the allowlist. On older npm use --ignore-scripts in CI with an explicit build step. pnpm: onlyBuiltDependencies. Yarn: enableScripts false." evidence: "The committed allowlist plus a CI install log showing scripts did not run for unapproved packages." enforce: "CI installs with scripts disabled; allowlist changes require review." - id: SUP-02 domain: D07 tier: P1 weight: 4 when: [always] title: "Lockfile is committed and CI installs are frozen and reproducible" risk: "A floating install lets a compromised version enter without any change to your repository." fix: "npm ci / pnpm install --frozen-lockfile / yarn --immutable / uv sync --frozen. CI must fail if the lockfile and manifest disagree." evidence: "CI log showing the frozen install command." enforce: "Frozen install is the only install path in CI." - id: SUP-03 domain: D07 tier: P1 weight: 3 when: [always] title: "A dependency cooldown / minimum release age is configured" risk: "In the March 2026 PyPI compromise, a malicious version was live for two and a half hours and was downloaded 119,000 times. A three-day delay would have caught it." fix: "npm min-release-age, pnpm minimumReleaseAge, Yarn npmMinimalAgeGate, pip --uploaded-prior-to, uv --exclude-newer. Dependabot applies a default cooldown; security updates bypass it, which is correct." evidence: "Config showing the cooldown value." enforce: "Set in the repo config and in the bot config." - id: SUP-04 domain: D07 tier: P1 weight: 3 when: [always] title: "Dependency vulnerability scanning runs in CI and on a schedule" risk: "Software Supply Chain Failures is A03:2025, new to the OWASP Top 10 and voted #1 risk by half of surveyed practitioners." fix: "Trivy, OSV-Scanner, Grype, npm audit, or Snyk. Scan on PR and nightly, since new CVEs land against unchanged code. Prioritize by exploitability (KEV, EPSS) not raw CVSS." evidence: "Scan output plus the triage decisions for anything unfixed." enforce: "Blocking on new critical/known-exploited; warning otherwise." - id: SUP-05 domain: D07 tier: P1 weight: 3 when: [always] title: "Every dependency actually exists and was not hallucinated" risk: "Slopsquatting. USENIX Security 2025 research over 576,000 samples found ~5% hallucination for commercial models and up to 21.7% for open models, with 43% of fabricated names reproduced consistently across runs — which is exactly what makes them registerable by attackers." fix: "For every dependency an AI added: confirm it is the package you meant, check publish date, download counts, repository link, and maintainer history. Be suspicious of any package created recently with a plausible name." evidence: "Review notes on newly added dependencies, or a CI check asserting each new dependency predates the PR." enforce: "New-dependency review is a required step in the PR template." - id: SUP-06 domain: D07 tier: P2 weight: 2 when: [always] title: "GitHub Actions are pinned to full commit SHAs" risk: "Tags are mutable. @v4 is a moving target controlled by someone else." fix: "Pin every third-party action to a full-length commit SHA with a version comment. Dependabot updates SHA pins in place." evidence: "Workflow files showing SHA pins." enforce: "Scorecard Pinned-Dependencies check, or a lint rule." - id: SUP-07 domain: D07 tier: P2 weight: 2 when: [always] title: "CI workflow token permissions are least-privilege and untrusted input is never interpolated into shell" risk: "Expression interpolation of a PR title into a run: block is remote code execution in your CI. pull_request_target with a checkout of the PR head is the classic total compromise." fix: "permissions: contents: read at workflow level, escalate per job. Pass untrusted event fields through env vars and quote them. Never check out untrusted code in a privileged workflow." evidence: "Workflow audit showing permissions blocks and no direct interpolation of event data." enforce: "Scorecard Token-Permissions and Dangerous-Workflow checks in CI." - id: SUP-08 domain: D07 tier: P2 weight: 2 when: [always] title: "An SBOM is generated per release and retained" risk: "When the next registry worm lands you need to answer 'are we affected' in minutes, not days. Enterprise and public-sector customers increasingly require it contractually." fix: "Generate CycloneDX or SPDX at build time from the build context with syft, trivy, or npm sbom. Store it with the release artifact. Attach as a signed attestation if you publish artifacts." evidence: "SBOM file attached to the most recent release." enforce: "SBOM generation step in the release workflow." - id: SUP-09 domain: D07 tier: P2 weight: 2 when: [always] title: "Published artifacts carry build provenance" risk: "Consumers cannot distinguish your build from an attacker's without attested provenance." fix: "Use platform attestations (e.g. actions/attest-build-provenance) or cosign keyless signing. If publishing to npm/PyPI, use trusted publishing (OIDC) rather than long-lived publish tokens." evidence: "Verification command output for the latest release." enforce: "Publishing without provenance is not possible in the release workflow." # ============================================================ # D08 - CI/CD & RELEASE ENGINEERING # ============================================================ - id: CI-01 domain: D08 tier: P0 weight: 4 when: [always] title: "Code is in version control with a clean, complete history" risk: "No repository means no rollback, no review, no history, no recovery." fix: "Git repository, meaningful commits, .gitignore covering env files, build output, and credentials. Lock files committed." evidence: "Repository URL and a clean git status." enforce: "N/A - foundational." - id: CI-02 domain: D08 tier: P1 weight: 4 when: [always] title: "CI runs lint, typecheck, tests, and build on every pull request" risk: "Without an automated gate, broken code reaches production at the speed of a merge button." fix: "One workflow running the full gate. Fast (<10 min) or people will bypass it." evidence: "A CI run link showing all steps green." enforce: "Required status checks configured on the protected branch." - id: CI-03 domain: D08 tier: P1 weight: 3 when: [has_team] title: "The main branch is protected: no direct pushes, no force pushes, review required" risk: "An unprotected main branch means any compromised credential or careless command rewrites production history." fix: "Branch protection or rulesets: require PR, require passing checks, require branches up to date, block force push and deletion, require Code Owner review on sensitive paths." evidence: "Screenshot or API output of the ruleset." enforce: "Ruleset applied; bypass list audited." - id: CI-04 domain: D08 tier: P1 weight: 3 when: [always] title: "CODEOWNERS protects auth, payments, migrations, and CI workflow paths" risk: "Most CI compromises in the 2025-2026 incident record involved modifying workflow files. Those paths deserve mandatory review." fix: ".github/CODEOWNERS assigning .github/workflows/, auth, payment, and migration directories to a named reviewer, combined with required Code Owner review." evidence: "CODEOWNERS file plus the ruleset requiring it." enforce: "Required Code Owner review on the protected branch." - id: CI-05 domain: D08 tier: P0 weight: 4 when: [is_public] title: "Deployments can be rolled back quickly, and rollback has been tested" risk: "Discovering that rollback does not work during an outage is the worst possible time to learn it." fix: "Document the exact rollback command or console action. Perform a rollback in a calm moment and time it. Account for migrations: a schema change may make a code rollback unsafe." evidence: "Dated rollback rehearsal with elapsed time and the command used." enforce: "Rollback procedure in the runbook, rehearsed at a defined cadence." - id: CI-06 domain: D08 tier: P1 weight: 3 when: [always] title: "Changes are previewable before production" risk: "Deploying straight to production is a coin flip that you win most of the time, which is what makes it dangerous." fix: "Branch preview deployments or a staging environment that mirrors production configuration." evidence: "A preview URL from a recent pull request." enforce: "Preview deploy runs automatically on every PR." - id: CI-07 domain: D08 tier: P1 weight: 3 when: [is_public] title: "Deploys are small, frequent, and attributable to a commit" risk: "Large batched deploys make attribution impossible: something broke, and it was one of forty changes." fix: "Deploy one meaningful change at a time. Tag releases. Record the deployed commit SHA in the app and in error tracking." evidence: "Release version visible in the running app and attached to error reports." enforce: "Build injects the commit SHA; error tracking receives release metadata." - id: CI-08 domain: D08 tier: P2 weight: 2 when: [is_public] title: "Risky changes ship behind feature flags with a kill switch" risk: "Without a flag, backing out a bad feature requires a deploy — minutes you may not have." fix: "A flag system (OpenFeature-compatible, or a simple config-backed one). Every risky feature gets an off switch reachable without a deploy. Clean up stale flags." evidence: "Flag definition plus a demonstration of toggling in production." enforce: "Flag lifecycle tracked; stale flags removed on a cadence." - id: CI-09 domain: D08 tier: P2 weight: 2 when: [is_public] title: "Post-deploy verification watches error rate, latency, and key funnels" risk: "A deploy that succeeds technically and breaks signup is still an outage." fix: "Define the watch window and metrics. Automate the rollback trigger if possible; otherwise a human watches for a defined period." evidence: "The post-deploy checklist plus a recent example of it being followed." enforce: "Automated canary analysis or alert thresholds tuned for post-deploy windows." # ============================================================ # D09 - ENVIRONMENTS, CONFIG & INFRASTRUCTURE # ============================================================ - id: ENV-01 domain: D09 tier: P0 weight: 4 when: [always] title: "Development, staging, and production are genuinely separate" risk: "Shared infrastructure means a test action has production consequences." fix: "Separate databases, storage buckets, API keys, queues, and third-party accounts (payment test mode, separate email sending domain or sandbox)." evidence: "Environment matrix listing each resource per environment." enforce: "Environment names asserted at boot; mismatched configuration refuses to start." - id: ENV-02 domain: D09 tier: P1 weight: 3 when: [always] title: "Required configuration is validated at startup and the app refuses to boot if it is wrong" risk: "A missing environment variable that silently defaults to a development value is a data-leak vector." fix: "Schema-validate all environment variables at boot. Fail loudly and immediately. No silent fallbacks for security-relevant settings." evidence: "Boot with a required variable removed; show the explicit failure." enforce: "Config schema in code; startup validation is unconditional." - id: ENV-03 domain: D09 tier: P1 weight: 3 when: [always] title: "Infrastructure changes are reproducible, not clicked" risk: "Console-configured infrastructure cannot be recreated after an account loss or a region failure, and nobody remembers what was changed." fix: "Infrastructure as code where practical, or at minimum a written inventory of every manually configured resource and setting." evidence: "IaC repository or the manual inventory document." enforce: "Drift detection, or a scheduled review of the inventory." - id: ENV-04 domain: D09 tier: P1 weight: 3 when: [is_public] title: "TLS everywhere, HTTP redirects to HTTPS, and certificate renewal is automated" risk: "An expired certificate is a total outage with a countdown you could have seen." fix: "HTTPS enforced, HSTS set, automated renewal, and an expiry monitor as a backstop." evidence: "SSL Labs or equivalent result plus the renewal mechanism." enforce: "Certificate expiry alert configured." - id: ENV-05 domain: D09 tier: P1 weight: 3 when: [always] title: "Serverless and platform timeouts are known and no core action exceeds them" risk: "Functions time out mid-write, leaving half-completed state and a 504 with no explanation." fix: "Document the timeout for each runtime. Measure p95 for every endpoint. Anything approaching the limit moves to a job." evidence: "Table of endpoint p95 latency against the platform timeout." enforce: "Alert when any endpoint p95 exceeds a fraction of the timeout." - id: ENV-06 domain: D09 tier: P2 weight: 2 when: [always] title: "DNS, domains, and registrar access are documented and secured" risk: "A lapsed domain or a hijacked registrar account is an unrecoverable brand and security event. Domain resurrection is a documented account-takeover vector." fix: "Auto-renew on, registrar lock enabled, MFA on the registrar account, ownership recorded in the handoff docs, expiry monitored." evidence: "Registrar settings and the documented owner." enforce: "Expiry monitor plus MFA enforced." # ============================================================ # D10 - OBSERVABILITY & ERROR TRACKING # ============================================================ - id: OBS-01 domain: D10 tier: P0 weight: 5 when: [is_public] title: "Error tracking is installed on both frontend and backend and receives production errors" risk: "OWASP A09:2025 is Security Logging and Alerting Failures. If an error happens and nothing records it, you learn about outages from angry users." fix: "Sentry, Rollbar, Bugsnag, or your platform's equivalent, wired into both runtimes, with source maps uploaded and release tagging enabled." evidence: "A deliberately triggered production error appearing in the dashboard with a readable stack trace." enforce: "Alert on new issue types and on error-rate spikes." - id: OBS-02 domain: D10 tier: P1 weight: 4 when: [always] title: "Logs are structured and carry a correlation ID across the request path" risk: "Unstructured logs are unsearchable at exactly the moment you need them. Without a request ID you cannot reconstruct what happened." fix: "JSON logs with timestamp, level, request/trace ID, route, user or account ID, and duration. Propagate the ID to background jobs and outbound calls." evidence: "A single request's log lines from entry through job completion, joined by one ID." enforce: "Logger wrapper enforces the shape; direct console logging is lint-blocked." - id: OBS-03 domain: D10 tier: P1 weight: 3 when: [always] title: "Logs redact secrets and personal data and have a defined retention period" risk: "Logs are the most commonly over-shared datastore in any company." fix: "Redaction at the logger. Retention set deliberately. Access restricted and reviewed." evidence: "Redacted sample plus the retention configuration." enforce: "Redaction unit-tested; retention configured at the provider." - id: OBS-04 domain: D10 tier: P1 weight: 4 when: [is_public] title: "Alerts exist for the failures that matter, and they reach a human" risk: "A dashboard nobody is watching at 2am is not monitoring." fix: "Alert on symptoms users feel: error-rate spike, latency at p95/p99, failed payments, failed jobs, queue depth, uptime failure, spend anomaly. Route to a channel that produces a notification. Every alert must be actionable — delete or tune anything that fires without requiring action." evidence: "Alert rule list plus proof one fired and reached someone." enforce: "Alert routing tested; on-call or notification path documented." - id: OBS-05 domain: D10 tier: P1 weight: 3 when: [is_public] title: "Uptime and synthetic monitoring check the real user journey, not just a 200" risk: "A health check that returns 200 while the database is down is worse than no health check." fix: "Separate liveness (is the process up) from readiness (are dependencies reachable). Add a synthetic check that performs a real login or core action on a schedule." evidence: "Monitor configuration plus an incident where it detected before a user did." enforce: "Monitors configured with alerting and reviewed." - id: OBS-06 domain: D10 tier: P2 weight: 3 when: [always] title: "Distributed tracing covers the slow and complex paths" risk: "Without traces, 'the app is slow' is an unanswerable question in a system with more than three hops." fix: "OpenTelemetry auto-instrumentation exporting to your backend. Instrument database calls, external APIs, jobs, and AI calls. Expect GenAI semantic conventions to keep changing; instrument anyway." evidence: "A trace for the slowest core endpoint showing where time goes." enforce: "Tracing enabled in production with a sampling policy." - id: OBS-07 domain: D10 tier: P2 weight: 2 when: [is_public] title: "Product analytics track the core funnel" risk: "Without funnel data, you optimize whatever you happen to notice." fix: "Instrument signup, activation, core action, and conversion. Respect consent requirements." evidence: "Funnel report for the last 30 days." enforce: "Events defined in a tracking plan; new features add events by default." # ============================================================ # D11 - RELIABILITY, BACKUP & INCIDENT RESPONSE # ============================================================ - id: REL-01 domain: D11 tier: P1 weight: 4 when: [is_public] title: "RTO and RPO are defined, written down, and consistent with the backup configuration" risk: "Everyone assumes recovery is fast until they measure it. Nightly backups mean up to 24 hours of data loss — decide that consciously." fix: "State the maximum tolerable downtime and data loss. Configure backups and failover to meet them. If they do not match, either change the configuration or change the promise." evidence: "RTO/RPO statement plus the measured restore time from DATA-02." enforce: "Reviewed when the architecture or the business promise changes." - id: REL-02 domain: D11 tier: P1 weight: 4 when: [is_public] title: "An incident runbook exists covering the realistic failure set" risk: "At 2am with users complaining, nobody invents a good process." fix: "Write procedures for: app down, database down, deploy gone bad, payment webhook failing, third-party provider outage, spend spike, secret leaked, suspected data exposure, restore from backup. Use references/templates/RUNBOOK.md." evidence: "Completed runbook with real commands, dashboard links, and provider support contacts." enforce: "Runbook reviewed after every incident." - id: REL-03 domain: D11 tier: P1 weight: 3 when: [is_public] title: "Someone is on call, or there is an explicit documented decision that nobody is" risk: "Ambiguity about who responds guarantees a slow response." fix: "Name who gets paged, how, and the expected response window. If the honest answer is 'best effort during business hours', write that down and set customer expectations accordingly." evidence: "Named owner and notification path, tested end to end." enforce: "Escalation path documented and rehearsed." - id: REL-04 domain: D11 tier: P1 weight: 3 when: [has_pii] title: "A breach and data-exposure response plan exists with notification timelines" risk: "GDPR requires notification to the supervisory authority within 72 hours of awareness. US state laws and sector rules impose their own clocks. You cannot draft this during the incident." fix: "Written plan: contain, assess scope, preserve evidence, determine notification obligations, notify, remediate, post-mortem. Identify who makes the notification call and which counsel to contact." evidence: "The plan, with named roles." enforce: "Reviewed annually and after any near-miss." - id: REL-05 domain: D11 tier: P2 weight: 2 when: [is_public] title: "Incidents get blameless post-mortems and produce concrete follow-up actions" risk: "Without them the same outage happens repeatedly." fix: "Timeline, impact, root cause, what made detection or recovery slow, action items with owners and dates. Use references/templates/INCIDENT-POSTMORTEM.md." evidence: "The most recent post-mortem, or a statement that no incidents have occurred." enforce: "Post-mortem is a required step for any user-visible incident." - id: REL-06 domain: D11 tier: P2 weight: 2 when: [is_public] title: "There is a way to tell customers what is happening" risk: "Silence during an outage costs more trust than the outage." fix: "Status page or a pre-agreed communication channel and a template message. Decide the threshold for posting." evidence: "Status page URL or the documented communication plan and template." enforce: "Included in the runbook's first steps." - id: REL-07 domain: D11 tier: P2 weight: 2 when: [has_third_party] title: "Third-party outage behavior is defined and degraded mode is tested" risk: "Your app's availability is the product of every dependency's availability unless you design otherwise." fix: "For each critical provider decide: fail closed, fail open, queue, or serve stale. Test by pointing the client at a failing endpoint." evidence: "Dependency table with the degraded behavior, plus a test showing it works." enforce: "Chaos-style dependency failure test in the suite." # ============================================================ # D12 - PERFORMANCE, CACHING & SCALE # ============================================================ - id: PERF-01 domain: D12 tier: P1 weight: 4 when: [always] title: "Database connection pooling is configured and the connection math works" risk: "Serverless plus a connection-per-invocation pattern exhausts Postgres connections at a traffic level far below what you expect. This is the most common first-scale failure." fix: "Use a pooler (PgBouncer, Supavisor, RDS Proxy, or your provider's). Compute: max concurrent instances x connections per instance must stay under the database limit with headroom." evidence: "The arithmetic, the configured pool size, and the database max_connections." enforce: "Alert on connection-pool saturation." - id: PERF-02 domain: D12 tier: P1 weight: 3 when: [always] title: "N+1 queries and unindexed slow queries have been found and fixed" risk: "ORM-generated N+1 patterns are invisible at ten rows and fatal at ten thousand. AI-generated data access is especially prone to them." fix: "Enable slow-query logging and pg_stat_statements or equivalent. Profile the top three pages. Add eager loading and the indexes the data demands." evidence: "Before/after query counts and timings for the heaviest endpoints." enforce: "Slow-query alert threshold configured; query-count assertions on hot paths." - id: PERF-03 domain: D12 tier: P1 weight: 3 when: [is_public] title: "Static assets are served from a CDN with correct cache headers" risk: "Serving assets from the application server wastes compute and produces a slow experience far from your region." fix: "Immutable, content-hashed asset filenames with long max-age. Short or no-cache for HTML. Image optimization and modern formats." evidence: "Response headers for a hashed asset and for an HTML document." enforce: "Set by the framework/platform configuration." - id: PERF-04 domain: D12 tier: P1 weight: 4 when: [is_public] title: "Private data is never cached where another user can receive it" risk: "A CDN or shared cache serving user A's dashboard to user B is a data breach caused by a cache header." fix: "Cache-Control: private, no-store on authenticated responses. Any cache key for user-specific data must include the user or tenant identity. Audit CDN rules for paths that bypass authentication." evidence: "Header inspection on an authenticated response plus a two-user cache test." enforce: "Test asserting no-store on authenticated routes." - id: PERF-05 domain: D12 tier: P2 weight: 3 when: [is_public] title: "A load test has been run at a realistic launch multiple and the breaking point is known" risk: "'It works for me' is a sample size of one. You need the number where it stops working." fix: "k6, Artillery, or Locust against staging. Ramp until something breaks. Record where latency degrades, where connections saturate, and where third-party limits trigger." evidence: "Load test report with the ramp profile and the identified breaking point." enforce: "Re-run before major launches and after architecture changes." - id: PERF-06 domain: D12 tier: P2 weight: 2 when: [always] title: "Expensive repeated work is cached with a defined invalidation strategy" risk: "Recomputing the same expensive result per request wastes money and latency; caching it wrong serves stale or wrong-user data." fix: "Cache read-heavy, slow, tolerably-stale results. Always define the invalidation trigger before adding the cache. Include tenant/user in the key when data is scoped." evidence: "Cache hit rate plus the documented invalidation rule." enforce: "Hit rate monitored." - id: PERF-07 domain: D12 tier: P2 weight: 2 when: [is_public] title: "A scaling plan exists for the next order of magnitude" risk: "Success is the most common outage cause." fix: "Identify the next bottleneck (connections, a single worker, an external rate limit, a queue) and write down what you would do. Autoscaling configured where available." evidence: "One page naming the next three bottlenecks and the response to each." enforce: "Revisited when traffic doubles." # ============================================================ # D13 - COST CONTROL & ABUSE PREVENTION # ============================================================ - id: COST-01 domain: D13 tier: P0 weight: 5 when: [has_ai] title: "No paid inference endpoint is publicly reachable without authentication and a rate limit" risk: "Unbounded Consumption is LLM10 in the OWASP LLM Top 10. A public AI endpoint is a credit card someone else is holding." fix: "Require authentication. Per-user and per-IP limits. A global circuit breaker that disables the feature above a spend threshold." evidence: "Call the endpoint unauthenticated (rejected) and hammer it authenticated (throttled). Show both." enforce: "Limits enforced at the gateway; breaker tested." - id: COST-02 domain: D13 tier: P0 weight: 4 when: [has_ai, has_third_party] title: "Hard spend caps and billing alerts exist at every paid provider" risk: "Providers will happily bill you for a runaway loop. The story of the $12,000 weekend is common and entirely preventable." fix: "Set provider-side hard limits where available and budget alerts everywhere. Alert at 50%, 80%, and 100% of expected monthly spend. Include hosting, database, storage, and egress, not just AI." evidence: "Screenshot or config of the caps and alert thresholds per provider." enforce: "Alerts route to a person; caps configured provider-side." - id: COST-03 domain: D13 tier: P1 weight: 4 when: [is_public] title: "Rate limiting exists on all public and expensive endpoints" risk: "Without limits, one script can exhaust your database connections, your third-party quota, and your budget simultaneously." fix: "Layered limits: per IP, per authenticated user, per account/tenant, and global. Return 429 with Retry-After. Apply at the edge where possible so the request never reaches your compute." evidence: "Load a public endpoint past the limit; show 429s." enforce: "Limits configured in edge/gateway config with tests." - id: COST-04 domain: D13 tier: P1 weight: 3 when: [has_ai, has_third_party] title: "Cost per user and per action is measured, not estimated" risk: "A unit economics surprise at scale is a business failure, not a technical one." fix: "Attribute every paid call to a user, account, and feature. Log tokens or units and an estimated cost. Build a cost-per-active-user figure and watch it." evidence: "The cost attribution table from references/15-ai-and-agents.md, populated with real numbers." enforce: "Cost dashboard exists and is reviewed." - id: COST-05 domain: D13 tier: P1 weight: 3 when: [is_public] title: "Abuse paths have been tested: signup spam, scraping, resource exhaustion" risk: "Free tiers, signup flows, and file processing are abused within days of any public launch." fix: "Bot protection or CAPTCHA on signup, email verification before resource-consuming actions, quotas on free tiers, size limits on anything processed." evidence: "Attempt each abuse path; document what stopped it." enforce: "Protections enabled with monitoring on signup anomalies." - id: COST-06 domain: D13 tier: P2 weight: 2 when: [has_payments] title: "Billing events reconcile with usage and entitlements" risk: "Silent revenue leakage: cancelled subscriptions still serving, failed renewals still granting access, usage not billed." fix: "Handle the full webhook set: created, updated, cancelled, payment failed, refunded, disputed. Test each in the provider's test mode. Reconcile entitlements against the provider as the source of truth." evidence: "Test-mode run of every subscription lifecycle event with the resulting entitlement state." enforce: "Automated reconciliation job comparing local entitlements to the provider." - id: COST-07 domain: D13 tier: P2 weight: 2 when: [always] title: "Infrastructure spend is forecast for the next 90 days at expected growth" risk: "Pricing cliffs (egress, function invocations, log ingestion, vector storage) surface at exactly the wrong moment." fix: "Model current cost per active user and project it. Identify which line item grows fastest and what the mitigation is." evidence: "A simple forecast table with the top three cost drivers." enforce: "Reviewed monthly." # ============================================================ # D14 - AI & AGENT SAFETY AND ECONOMICS # ============================================================ - id: AI-01 domain: D14 tier: P0 weight: 5 when: [has_ai] title: "Model provider calls happen server-side only, through a single gateway module" risk: "A provider key in client code is stolen immediately and billed to you. A scattered call pattern makes quotas, caching, and cost tracking impossible." fix: "One provider wrapper. All calls route through it. Authentication, quota enforcement, cost logging, retry, fallback, and telemetry live there." evidence: "Grep for direct SDK usage outside the wrapper and for keys in the client bundle; both empty." enforce: "Lint rule blocking direct SDK imports; bundle scan in CI." - id: AI-02 domain: D14 tier: P0 weight: 5 when: [has_agents] title: "The lethal trifecta is broken: no single agent context combines private data, untrusted content, and an external communication channel" risk: "Prompt injection is unsolved. Independent 2025-2026 evaluations found over 90% attack success against published defenses under adaptive attack. The only reliable protection is architectural: deny the attacker a path to exfiltrate." fix: "Apply Meta's Rule of Two — at most two of (untrusted input, sensitive data access, ability to change state or communicate externally) without human supervision. Egress allowlist. Separate the agent that reads untrusted content from the agent holding credentials. Consider plan-then-execute, dual-LLM, or map-reduce isolation patterns." evidence: "A data-flow diagram per agent workflow marking which of the three properties it holds and how the third is prevented." enforce: "Egress allowlist enforced at the network layer; architecture reviewed when tools are added." - id: AI-03 domain: D14 tier: P0 weight: 4 when: [has_agents] title: "Agent tools are individually scoped and no agent holds production credentials" risk: "The best-documented AI coding disaster deleted a production database. The root cause was not the deletion — it was that the agent had production credentials at all. Instructions in a prompt are not a security control." fix: "Each tool gets its own least-privilege credential. No blanket database or shell access. Read and write are separate tools. Destructive operations require an out-of-band confirmation the model cannot issue itself." evidence: "Tool inventory listing each tool, its credential, its blast radius, and whether it requires approval." enforce: "Credentials issued per tool by infrastructure, not shared from a single environment variable." - id: AI-04 domain: D14 tier: P1 weight: 4 when: [has_agents] title: "Irreversible and externally visible actions require human approval with full parameter visibility" risk: "An approval prompt that says 'Allow?' without showing the payload trains users to click yes. Approval fatigue is itself a documented attack technique." fix: "Show exactly what will happen — recipient, amount, target, full command. Rate-limit approval requests so flooding cannot exhaust attention. Log every approval and denial." evidence: "Screenshot of an approval prompt showing complete parameters." enforce: "Approval is enforced server-side, not by the model choosing to ask." - id: AI-05 domain: D14 tier: P1 weight: 3 when: [has_ai] title: "Model output is treated as untrusted input everywhere it is used" risk: "LLM05 Improper Output Handling. Model output rendered as HTML is XSS; passed to a shell is command injection; passed to SQL is injection; used in a redirect is an open redirect." fix: "Validate and encode model output at every sink exactly as you would user input. Parse structured output against a schema. Never eval, never interpolate into a shell." evidence: "Inventory of output sinks and the validation applied at each." enforce: "Schema validation on structured outputs; SAST rules on the sinks." - id: AI-06 domain: D14 tier: P1 weight: 3 when: [has_ai] title: "Token usage and cost are logged per request with full attribution" risk: "Without attribution you cannot find the expensive feature, the abusive user, or the runaway loop." fix: "Log provider, model, feature, user, account, input tokens, output tokens, cached tokens, latency, and estimated cost for every call." evidence: "Sample log lines plus an aggregate by feature." enforce: "Emitted by the gateway wrapper for every call." - id: AI-07 domain: D14 tier: P1 weight: 3 when: [has_ai] title: "Per-user and per-tier quotas are token-budgeted, not request-counted" risk: "One agentic user turn can fan out to hundreds of model calls. Request counting is meaningless for agents." fix: "Token budgets per user per period, separate ceilings for input and output, a per-session cost cap, and hard limits on agent loop iterations and tool-call depth." evidence: "Quota configuration plus a test showing enforcement and a graceful message at the limit." enforce: "Enforced in the gateway; breach alerts." - id: AI-08 domain: D14 tier: P1 weight: 3 when: [has_ai] title: "Provider failure degrades gracefully" risk: "Model providers have outages and rate limits. An unhandled 429 or 503 becomes a broken product." fix: "Timeouts, bounded retries with backoff, a fallback model or provider, and a clear user-facing message or queued retry. Never silently return an empty result." evidence: "Simulate a provider failure; show the degraded behavior." enforce: "Fallback path covered by a test." - id: AI-09 domain: D14 tier: P1 weight: 3 when: [has_ai] title: "An eval set exists for the AI feature and runs before releases" risk: "Prompt and model changes are untestable by intuition. Non-determinism means a change that looks fine can degrade a whole category of inputs." fix: "Build a golden set of 50-150 real examples from actual usage, weighted toward observed failure modes. Prefer deterministic assertions and code-based checks over LLM judges. If you use a judge, validate it against human labels and keep it a scoped binary question." evidence: "The eval set, the pass criteria, and a run result." enforce: "Evals run in CI on any prompt, model, or retrieval change." - id: AI-10 domain: D14 tier: P1 weight: 3 when: [has_ai] title: "Prompt injection has been tested against your actual attack surface" risk: "Every place untrusted content enters the model context — documents, web pages, emails, tickets, tool output, other users' content — is an injection surface. Tool schemas are injection surface too, not just descriptions." fix: "Build an injection test suite reflecting your real inputs. Test instruction override, data exfiltration attempts, tool misuse, and system prompt extraction. Track the success rate as a metric across releases." evidence: "Test suite with attempt/result and a measured success rate." enforce: "Injection suite runs in CI; success rate tracked over time." - id: AI-11 domain: D14 tier: P2 weight: 3 when: [has_ai] title: "Cost is engineered: caching, routing, and batching where they apply" risk: "Sending every request to the largest model with a cold cache is typically 5-20x more expensive than necessary." fix: "Prompt caching on stable prefixes — verify you exceed the model's minimum cacheable size and that no per-request value sits inside the cached prefix, which silently zeroes hit rate. Route simple requests to cheaper models. Use batch APIs for anything not latency-sensitive. Cap output tokens, since output is the dominant cost term." evidence: "Cache hit rate, model distribution by tier, and the before/after cost per action." enforce: "Cache hit rate and cost per action monitored." - id: AI-12 domain: D14 tier: P2 weight: 2 when: [has_ai] title: "AI behavior is traced and reviewed against real production traffic" risk: "Failure modes you have not seen cannot be in your eval set. Error analysis on real traces is the highest-value evaluation activity." fix: "Trace every AI request with inputs, outputs, tool calls, latency, and cost. Sample and manually review 20-100 traces on a regular cadence; group failures into a taxonomy; promote recurring failures into the eval suite." evidence: "Trace samples plus a failure taxonomy with counts." enforce: "Review cadence scheduled with an owner." - id: AI-13 domain: D14 tier: P2 weight: 2 when: [has_ai] title: "AI features are disclosed to users where required and outputs are labelled" risk: "EU AI Act Article 50 transparency obligations apply to systems that interact with people or generate synthetic content. Verify the current status and dates against primary sources before relying on any timeline." fix: "Disclose that the user is interacting with an AI system. Mark AI-generated or manipulated content. Document the disclosure in your privacy policy." evidence: "Screenshot of the disclosure in product." enforce: "Part of the release checklist for AI features." - id: AI-14 domain: D14 tier: P2 weight: 2 when: [has_agents] title: "Third-party tool and MCP server integrations are vetted and pinned" risk: "Tool poisoning, rug pulls where a server changes its tool definitions after approval, and cross-server shadowing are documented attack classes with real incidents including a CVSS 9.6 remote code execution in a popular MCP proxy." fix: "Maintain a registry of approved servers/tools. Pin tool definitions by hash and alert on drift. Run local servers sandboxed. Require OAuth with audience-bound tokens rather than shared credentials. Human approval for adding any tool." evidence: "Approved tool registry with pinned definition hashes." enforce: "Drift detection alerting on tool definition changes." - id: AI-15 domain: D14 tier: P2 weight: 2 when: [has_agents] title: "Agent memory and retrieved context cannot be poisoned across users or sessions" risk: "Memory poisoning is a top-ranked agentic threat. Content written by one user that persists into another user's context is a cross-user injection channel." fix: "Scope memory and vector stores by user or tenant in the retrieval filter, not just at write time. Treat anything retrieved as untrusted. Expire memory. Store only what changes future behavior." evidence: "Test: user A stores an injection payload; user B's retrieval does not surface it." enforce: "Tenant filter enforced in the retrieval layer with a test." # ============================================================ # D15 - PRIVACY, LEGAL & COMPLIANCE # ============================================================ # NOT LEGAL ADVICE. Verify current requirements and dates against primary # sources and qualified counsel. Regulations in this domain change frequently. - id: LEG-01 domain: D15 tier: P0 weight: 4 when: [has_pii] title: "A privacy policy and terms of service are published and accurate" risk: "Collecting personal data without a privacy notice is a violation in most jurisdictions and blocks app store review." fix: "Publish both. The privacy policy must actually describe what you collect, why, who you share it with (list your processors), how long you keep it, and how users exercise their rights. A generic template that misdescribes your processing is worse than none." evidence: "Live URLs plus a mapping from each policy claim to the actual system behavior." enforce: "Reviewed when data flows change." - id: LEG-02 domain: D15 tier: P0 weight: 4 when: [has_pii] title: "A data inventory exists: what you collect, where it lives, who it goes to" risk: "You cannot honor a deletion request, answer a security questionnaire, or assess breach scope without knowing where data is." fix: "Table of data element, purpose, store, retention, and every third party that receives it (analytics, error tracking, AI providers, email, support tools). Error trackers and AI providers are the most commonly forgotten processors." evidence: "The completed inventory." enforce: "Updated when a new vendor or data field is added; part of the PR template for schema changes." - id: LEG-03 domain: D15 tier: P1 weight: 3 when: [has_pii] title: "Data subject rights are actually operable: access, deletion, correction, export" risk: "Rights requests carry statutory deadlines. A manual process you have never run will miss them." fix: "A documented procedure, ideally self-service in product. Must reach every store including backups (with a stated expiry window), analytics, and downstream processors." evidence: "Execute each request type against a test account end to end; document elapsed time." enforce: "Self-service in product, or a tracked queue with SLA monitoring." - id: LEG-04 domain: D15 tier: P1 weight: 3 when: [eu_users] title: "GDPR basics are in place: lawful basis, processor agreements, transfer mechanism, cookie consent" risk: "Processor agreements with every vendor handling personal data are mandatory, as is a valid international transfer mechanism." fix: "Identify the lawful basis per processing purpose. Execute a DPA with every processor. Confirm your transfer mechanism. Cookie/tracker consent must be genuine opt-in for non-essential trackers, with refusal as easy as acceptance. Verify current requirements — this area moves." evidence: "DPA list, lawful basis table, consent banner behavior verified with a network trace showing no non-essential trackers fire before consent." enforce: "Consent gating enforced in code; new vendors require a DPA." - id: LEG-05 domain: D15 tier: P1 weight: 3 when: [us_state_privacy] title: "US state privacy obligations are handled, including universal opt-out signals" risk: "The number of US states with comprehensive privacy laws has grown substantially and several require honoring browser-level opt-out signals such as Global Privacy Control. Verify which states apply to you and their current thresholds." fix: "Determine applicability by your consumer counts and revenue. Implement opt-out of sale/sharing, honor GPC, publish the required notices, and handle sensitive-data limits." evidence: "Applicability analysis plus a test showing GPC is detected and honored." enforce: "GPC handling covered by a test." - id: LEG-06 domain: D15 tier: P1 weight: 3 when: [has_payments] title: "Payment card scope is minimized and the applicable PCI obligations are known" risk: "Embedding payment fields on your own page rather than using a hosted field or redirect materially expands your PCI scope, including requirements around managing and monitoring scripts on payment pages. Verify current PCI DSS requirements." fix: "Prefer a hosted checkout or the provider's iframe/hosted fields so card data never touches your servers or DOM. Never log or store card numbers. Confirm which self-assessment questionnaire actually applies to your integration." evidence: "Integration type documented, plus confirmation no card data touches your systems or logs." enforce: "CSP restricting scripts on payment pages; script inventory maintained." - id: LEG-07 domain: D15 tier: P1 weight: 3 when: [has_payments] title: "Subscription, refund, and cancellation terms are clear and cancellation is easy" risk: "Consumer protection regulators and app stores both take an interest in hard-to-cancel subscriptions. Requirements vary by jurisdiction and change; verify current rules." fix: "Disclose price, renewal cadence, and cancellation method before purchase. Make cancellation available through the same channel as signup. Send renewal reminders where required." evidence: "Screenshots of the disclosure and the cancellation flow." enforce: "Part of the release checklist for billing changes." - id: LEG-08 domain: D15 tier: P2 weight: 2 when: [is_regulated] title: "Sector-specific obligations are identified and the controls mapped" risk: "HIPAA, FERPA, GLBA, SOC 2, and similar regimes impose specific controls. Discovering them during an enterprise sales cycle costs months." fix: "Identify the regime, obtain necessary agreements (e.g. a BAA before any regulated data flows), and map each required control to a specific implementation. Compliance automation platforms help but do not substitute for the controls." evidence: "Control mapping document." enforce: "Evidence collection automated where possible; owner assigned." - id: LEG-09 domain: D15 tier: P2 weight: 2 when: [is_public] title: "Accessibility obligations have been assessed for your markets" risk: "Accessibility requirements now carry legal force in multiple jurisdictions with staggered deadlines and exemptions that may or may not apply to you. Verify current scope and dates." fix: "Determine which regimes apply. WCAG 2.2 Level AA is the practical target in most cases. Publish an accessibility statement if required." evidence: "Applicability note plus the current conformance status from FE-05 and FE-06." enforce: "Accessibility checks in CI; statement maintained." - id: LEG-10 domain: D15 tier: P2 weight: 2 when: [has_ugc] title: "Content moderation, reporting, and takedown paths exist" risk: "Any surface where users publish to other users attracts abuse, illegal content, and copyright claims. Platform obligations vary by jurisdiction and scale." fix: "A reporting mechanism, a defined review process, a takedown path, and terms that permit removal. Copyright agent designation where applicable." evidence: "Reporting flow and the documented review process." enforce: "Reports route to a monitored queue with an owner." # ============================================================ # D16 - TESTING & VERIFICATION # ============================================================ - id: TEST-01 domain: D16 tier: P0 weight: 5 when: [always] title: "Automated tests cover authorization on every endpoint" risk: "This is the single highest-value test suite in an AI-built codebase, because missing authorization is the most common and most damaging AI omission." fix: "Parameterized test iterating every endpoint x (anonymous, wrong user, wrong tenant, wrong role) asserting the correct rejection." evidence: "The test file and a passing run showing the endpoint coverage count." enforce: "Test fails when a new endpoint is added without an entry." - id: TEST-02 domain: D16 tier: P1 weight: 4 when: [always] title: "The critical user journeys have end-to-end tests" risk: "Unit tests pass while the product is broken, because AI-generated code fails at wiring rather than at logic." fix: "Playwright (or equivalent) covering the three to six journeys that would put you on the phone with customers. Seed data via API or direct database access, reuse authenticated state, never drive login through the UI except in the login test." evidence: "Test run output listing the covered journeys." enforce: "Smoke subset runs on every PR; full suite on merge and nightly." - id: TEST-03 domain: D16 tier: P1 weight: 4 when: [has_payments] title: "The full payment lifecycle is tested in the provider's test mode" risk: "Payment bugs are the most expensive category: they lose money invisibly and destroy trust." fix: "Test success, decline, 3DS challenge, webhook signature verification, duplicate webhook delivery, subscription cancellation, failed renewal, refund, dispute, and double-submit idempotency." evidence: "Completed payment test matrix with results." enforce: "Automated where the provider supports it; manual matrix re-run before billing changes." - id: TEST-04 domain: D16 tier: P1 weight: 3 when: [always] title: "Tests actually assert; coverage is measured on the diff, not chased globally" risk: "AI-generated tests are disproportionately prone to executing code with vacuous assertions, inflating coverage while detecting nothing." fix: "Review AI-written tests for real assertions. Gate on coverage of changed lines rather than a global percentage. Run mutation testing (e.g. Stryker) once as an audit of the existing suite to find the hollow tests." evidence: "Diff coverage report plus a mutation score for the core modules." enforce: "Diff coverage threshold as a required check." - id: TEST-05 domain: D16 tier: P1 weight: 3 when: [is_public] title: "The API has been fuzzed against its schema" risk: "Edge-case inputs producing 500s, responses that drift from the documented contract, and validation that accepts invalid data are exactly the AI-generated handler failure profile." fix: "Run a schema-driven fuzzer (e.g. Schemathesis) against your OpenAPI spec. It requires no test authoring and finds this class immediately." evidence: "Fuzzer run output and the issues fixed." enforce: "Fuzz run in CI or nightly against staging." - id: TEST-06 domain: D16 tier: P1 weight: 3 when: [always] title: "Tests run against a real database, not mocks, for data-layer behavior" risk: "Mocked database tests validate your mocks. Constraint violations, transaction behavior, and RLS policies only appear against the real engine." fix: "Testcontainers or a dedicated test database. Isolate via per-test transactions, template databases, or per-worker schemas so tests can run in parallel." evidence: "Test setup showing the real engine, plus a parallel run." enforce: "CI provisions the database automatically." - id: TEST-07 domain: D16 tier: P2 weight: 2 when: [always] title: "Flaky tests are quarantined and fixed, not retried into silence" risk: "A suite that fails randomly gets ignored, and then it is worse than no suite." fix: "Retries with trace capture on first retry so you can actually diagnose. Tag persistent flakes into a non-blocking project and fix them on a schedule. Never leave a permanently red check." evidence: "Flake list with owners and the trend." enforce: "Flake rate tracked; quarantine has an expiry." - id: TEST-08 domain: D16 tier: P2 weight: 2 when: [always] title: "A human has read the security-critical code, not just the tests" risk: "METR's randomized trial found experienced developers were 19% slower with AI assistance while believing they were 20% faster — self-assessment of AI-assisted work is demonstrably unreliable. External verification is required." fix: "Line-by-line human review of auth, authorization, payment, migration, and any code touching secrets. Use references/21-ai-code-failure-modes.md as the review checklist." evidence: "Review record naming the reviewer, the files, and the date." enforce: "CODEOWNERS requires review on those paths." # ============================================================ # D17 - MOBILE & APP STORE # ============================================================ # Store policies change frequently. Verify every requirement against the # current Apple App Store Review Guidelines and Google Play policy pages # before submission; treat the items below as the checklist, not the spec. - id: MOB-01 domain: D17 tier: P0 weight: 4 when: [has_mobile] title: "Account deletion is available in-app for any app that supports account creation" risk: "This is a standing Apple review requirement and a Google Play policy requirement. It is one of the most common avoidable rejections." fix: "In-app initiation of full account and data deletion, not merely deactivation, not email-only. Provide a web-accessible deletion path where required." evidence: "Screen recording of the in-app deletion flow completing." enforce: "Part of the release checklist." - id: MOB-02 domain: D17 tier: P0 weight: 4 when: [has_mobile] title: "Privacy declarations match actual data collection, including every SDK" risk: "Apple privacy labels and manifests and Google Play's Data safety form must reflect reality including third-party SDK behavior. Mismatches cause rejection and, later, enforcement." fix: "Enumerate every SDK and what it collects. Complete the privacy declarations honestly. Provide required-reason API declarations and SDK privacy manifests where mandated. Verify current requirements with Apple and Google directly." evidence: "SDK inventory mapped to each declared data category." enforce: "Inventory updated whenever a dependency is added." - id: MOB-03 domain: D17 tier: P0 weight: 3 when: [has_mobile] title: "Monetization complies with current store rules" risk: "Payment rules — in-app purchase requirements, external link entitlements, and anti-steering provisions — have changed repeatedly through litigation and vary by region. Assumptions here get apps removed." fix: "Determine which purchase mechanism your content requires. Verify current rules for your regions directly from the store policies before shipping. Implement receipt validation server-side, never client-side." evidence: "Policy determination note plus server-side receipt validation code." enforce: "Reviewed before every release that touches monetization." - id: MOB-04 domain: D17 tier: P1 weight: 3 when: [has_mobile] title: "Target SDK/API level and platform technical requirements meet current store minimums" risk: "Both stores enforce rolling minimum SDK/API levels and periodic technical requirements. Missing a deadline blocks updates entirely — including security fixes." fix: "Check the current required target API level and build SDK for both stores, and any active platform requirements. Calendar the next deadline." evidence: "Build configuration values plus the current published requirement." enforce: "Calendar reminder ahead of each announced deadline." - id: MOB-05 domain: D17 tier: P1 weight: 3 when: [has_mobile] title: "A forced-update mechanism exists" risk: "Without one, a client-side bug or an API breaking change strands users on a broken version indefinitely." fix: "Server-driven minimum-version check that blocks the app with an update prompt. Version your API so old clients degrade rather than break." evidence: "Demonstrate the block by lowering the client version." enforce: "Minimum version is a server-side configuration value." - id: MOB-06 domain: D17 tier: P1 weight: 3 when: [has_mobile] title: "Crash reporting and release-tagged error tracking are live on mobile" risk: "Mobile crashes are invisible without instrumentation, and store review surfaces crash rate." fix: "Crashlytics, Sentry, or equivalent with symbolication and release tagging. Watch crash-free-session rate per release." evidence: "Dashboard showing crash-free rate for the current release." enforce: "Alert on crash-rate regression after release." - id: MOB-07 domain: D17 tier: P1 weight: 3 when: [has_mobile] title: "Secrets are not embedded in the app binary and the API does not trust the client" risk: "Mobile binaries are trivially decompiled. Anything shipped is public. API keys in an app are public keys." fix: "No privileged keys in the binary. Every mobile-facing API endpoint enforces authentication and authorization independently. Use platform attestation (App Attest / Play Integrity) if you need to limit access to genuine clients — but never as the only control." evidence: "String extraction on the built binary showing no privileged secrets." enforce: "Binary secret scan in the release pipeline." - id: MOB-08 domain: D17 tier: P1 weight: 2 when: [has_mobile] title: "Sensitive data uses platform secure storage, and offline behavior is defined" risk: "Tokens in plain preferences or unencrypted local databases are readable on a compromised device." fix: "Keychain / Keystore for credentials. Define what works offline, how conflicts resolve, and what the app shows with no connectivity." evidence: "Storage code plus an airplane-mode walkthrough of core screens." enforce: "Covered by device tests." - id: MOB-09 domain: D17 tier: P2 weight: 2 when: [has_mobile] title: "Deep links and universal links are verified and handle unauthenticated and cold-start cases" risk: "A broken universal link silently falls back to the browser and breaks every email and notification flow." fix: "Verify the association files. Test cold start, warm start, logged out, and invalid target for each link type." evidence: "Link test matrix with results." enforce: "Included in the release checklist." - id: MOB-10 domain: D17 tier: P2 weight: 2 when: [has_mobile] title: "Push notification credentials and their expiry are tracked, and rollout is staged" risk: "Expired push credentials silently kill notifications. A full-population release with no staged rollout has no abort option." fix: "Record credential expiry dates with calendar reminders. Use phased/staged rollout on both stores and define the halt criteria." evidence: "Credential expiry register plus the staged rollout configuration." enforce: "Expiry monitored; staged rollout is the default release mode." # ============================================================ # D18 - DOCUMENTATION & HANDOFF # ============================================================ - id: DOC-01 domain: D18 tier: P1 weight: 4 when: [always] title: "README lets a new developer run the project locally from zero" risk: "Undocumented setup is the most common reason a project stalls when the original builder is unavailable." fix: "Prerequisites, install, environment variable names (never values), database setup and seed, run, test, and common problems. Verify by following it on a clean machine or container." evidence: "Someone other than the author followed it successfully, or a clean-container run of the documented steps." enforce: "Setup steps exercised by CI (the CI workflow is the executable README)." - id: DOC-02 domain: D18 tier: P1 weight: 3 when: [always] title: "Architecture notes exist: system diagram, data model, vendor map, auth model" risk: "Without them, every change requires re-deriving the system from source, which is where regressions come from." fix: "Use references/templates/ARCHITECTURE.md. Include the request path, the data model, every third-party vendor and what it does, the permission model, background jobs, and where money and personal data flow." evidence: "The completed document." enforce: "Updated as part of any architectural change PR." - id: DOC-03 domain: D18 tier: P1 weight: 3 when: [is_public] title: "Deploy and rollback procedures are written down and executable by someone else" risk: "If only one person can deploy, that person is a single point of failure for every incident." fix: "Exact commands or console steps, environment by environment, including the migration procedure and the rollback path." evidence: "Another person executed a deploy following only the document." enforce: "Deployment is automated; the document describes the pipeline and the manual override." - id: DOC-04 domain: D18 tier: P2 weight: 2 when: [has_team] title: "Significant decisions are recorded with their reversal path" risk: "Six months later nobody remembers why the odd choice was made, so it gets undone and the original problem returns." fix: "A short ADR per significant decision: context, decision, alternatives, consequences, reversal plan. Use references/templates/ADR.md." evidence: "ADR directory with entries for the major choices." enforce: "ADR required for architectural PRs." - id: DOC-05 domain: D18 tier: P2 weight: 2 when: [always] title: "Known limitations, deliberate shortcuts, and technical debt are listed honestly" risk: "Undocumented shortcuts become production incidents that surprise everyone including the person who took them." fix: "A LIMITATIONS section listing what is not handled, what will break at scale, and what was deliberately deferred with the trigger for revisiting it." evidence: "The list." enforce: "Reviewed each FINISHER run." # ============================================================ # D19 - PRODUCT TRUTH & OUTCOME MEASUREMENT # ============================================================ - id: PROD-01 domain: D19 tier: P1 weight: 4 when: [always] title: "The problem, the user, and the success metric are written down in one paragraph" risk: "A technically perfect product solving a problem nobody has is still a failure. This is the check most engineering checklists omit." fix: "One paragraph: who, what problem, what they do today instead, and the single number that proves it worked." evidence: "The paragraph, with the metric named and its current value." enforce: "Revisited each FINISHER run." - id: PROD-02 domain: D19 tier: P1 weight: 3 when: [always] title: "A baseline was captured before changes so improvement is provable" risk: "Without a before, every after is an anecdote." fix: "Record time on task, error rate, volume handled, cost per task, completion rate, and support burden before optimizing." evidence: "The baseline table with the date it was captured." enforce: "Baseline recorded in the FINISHER ledger at run 1." - id: PROD-03 domain: D19 tier: P2 weight: 3 when: [is_public] title: "Real user outcomes are measured after launch, not just system health" risk: "Green dashboards and zero adoption is the most common launch outcome for a technically sound product." fix: "Track activation, retention, and the core success metric. Set a review date and a decision rule for what you do if it is not met." evidence: "Post-launch metric report against the target." enforce: "Scheduled review with a named owner." - id: PROD-04 domain: D19 tier: P2 weight: 2 when: [is_public] title: "There is a way for users to report problems and it is monitored" risk: "Users who cannot tell you it is broken simply leave." fix: "A support address, in-app feedback, or a form, routed somewhere a human reads with a stated response expectation." evidence: "The channel and its owner." enforce: "Response time tracked." ```