ProcureCheck
ProcureCheck is a PRD for an internal RAG assistant that helps procurement managers at German manufacturing firms check supplier contracts against REACH/CE/GDPR compliance requirements — cutting a 2–3 day legal-escalation bottleneck down to same-day, clause-cited answers. It's designed to refuse rather than guess: no legal sign-offs, no answers on un-ingested suppliers, and every claim traceable to a specific clause. The PRD covers architecture justification, failure-mode analysis, cost modeling, and a phased launch plan with go/no-go metrics.
Situation
Procurement managers and legal liaisons at 200–1000-person German manufacturing/industrial companies review 15–20 supplier contracts a month, and roughly 1 in 5 of those hit a compliance ambiguity — a REACH, CE, or GDPR data-processing clause that isn't a clean yes/no. Today's workflow is entirely manual and scattered across five slow, error-prone steps: contracts arrive buried in inboxes with no central intake; the relevant compliance checklist has to be manually located in SharePoint, where outdated copies linger in local folders; the contract is read clause-by-clause against that checklist with no search tooling; any ambiguous clause is emailed to legal and waits 2–3 days, blocking supplier onboarding; and the outcome is logged in an Excel tracker with no audit trail linking the decision back to the source clause.
Task
Design a product requirements document for an internal RAG assistant — ProcureCheck — that checks supplier contracts and certifications against the company's own compliance requirements, grounded in its actual contract repository rather than the model's general knowledge, and cites the exact clause behind every answer. Explicit constraint I set for myself: the system must never give a definitive legal sign-off, and must never guess about a supplier or contract it hasn't ingested. Given the liability at stake — REACH, CE, and GDPR compliance decisions — a confident-sounding wrong answer is worse than a system that visibly says “I don't know” and hands off to legal.
Action
1. Justified the architecture choice explicitly: ruled out better prompting alone (the model has no access to the company's actual contracts or checklists) and fine-tuning (contracts and regulations update quarterly, and fine-tuning can't cite the exact clause an answer came from) before settling on RAG, which retrieves the current checklist and contract at query time and can cite the exact paragraph. 2. Prioritized ruthlessly with justification, not just a P0/P1 label: onboarding checks and regulation-update re-checks as P0 (highest frequency and highest risk respectively), explicit low-confidence signaling as P0 (“the difference between a useful tool and a liability”), and smart escalation-routing deferred to P1 in favor of a simpler always-offer-escalation pattern for v1. 3. Made clause-aware chunking a first-class architecture decision: splitting on clause/section boundaries (~300–500 tokens) with clause number kept as metadata, because a compliance answer needs a whole clause, not half of one — then paired it with an EU-hosted embedding model and a mid-tier EU-hosted LLM (Mistral Small/Large) to satisfy data residency end-to-end without over-paying for reasoning the task doesn't need. 4. Enumerated six concrete failure modes before launch — stale checklist copies, chunking misfires on non-standard templates, retrieval misses on regulation synonyms, LLM overclaiming beyond the cited clause, approvals going stale after a regulation update, and users mistaking a match for a legal sign-off — and paired each with a detection method and a fix, rather than a generic “we'll monitor for issues.” 5. Designed the eval loop around defensibility, not convenience: retrieval precision@5, faithfulness (does the cited clause actually contain the claimed content, auto-checkable), and escalation rate as core metrics, with human review — not LLM-as-judge alone — grading the eval set before every release, because a wrong compliance answer has real cost. 6. Chose distribution based on where the user already lives: a Microsoft Teams bot rather than a standalone web app, because German manufacturing/industrial companies overwhelmingly run on Microsoft 365 and procurement managers already work in Teams/SharePoint — removing an adoption barrier a new destination wouldn't. 7. Wrote a full system-prompt build brief and ran it through three test interactions (happy path, refusal on an un-ingested supplier, and an escalation request) rather than stopping at the spec — and documented, honestly, where the model did more than the brief required (unprompted-but-useful GDPR flagging) versus exactly what was specified, distinguishing lucky behavior from guaranteed behavior.
Result
1. A complete PRD with a phased launch plan — internal alpha (5–8 managers, gated on ≥80% retrieval precision@5 and zero unflagged hallucinated citations) → closed beta (full procurement team + legal, gated on stable/declining escalation rate and sub-1-day resolution) → GA (company-wide, gated on audit-log completeness verified by a real compliance audit dry-run). 2. A cost model showing the architecture is viable at scale: roughly $0.0005 per query and ~$1/month in total LLM cost at 2,000 queries/month — with the honest observation that vector DB hosting and document ingestion, not token cost, is the actual dominant cost driver worth optimizing. 3. A build brief that was actually tested, not just written: three example interactions run against the system prompt, each scored against the MUST requirements (cite every claim, refuse to guess on un-ingested suppliers, decline legal sign-off and route to legal) with gaps and over-performance both called out explicitly. 4. A permissions and residency model addressing real deployment constraints: role-based chunk filtering (a junior analyst can't see commercial pricing terms even where they can see compliance clauses) and an EU-region requirement for both the vector DB and document store, not just the embedding step. 5. A refusal-scoping decision list that keeps the product trustworthy rather than merely capable — explicit statements of what ProcureCheck should refuse to do (definitive legal sign-off, answering on non-ingested contracts) built in from the PRD stage rather than discovered after a bad answer shipped.
Closing Notes
This PRD isn't a claim that RAG solves compliance review on its own — it's a demonstration of how I approach AI product design end to end: justify the architecture against real alternatives, prioritize by risk and frequency rather than instinct, plan for failure before it happens, and hold the system to a defensible evaluation standard rather than a demo-friendly one. If you're interested, I'd welcome the chance to walk through the reasoning behind any of these decisions.