Reasoning Audit Layer · B2B adoption brief · v0.1
After the Answer Comes Back
The design space for effectiveness evidence in a reasoning audit layer
The evidence chain in governance products stops at what the organisation said. Nobody produces evidence of what the user received. The regulation actually asks for the latter — Art. 50 says clear and distinguishable, Law 25 says informed, PIPEDA says meaningful. Those three adjectives describe something that happens in a person's head. A checklist cannot answer that. A document cannot either.
This brief splits the audit layer into four insertion points in a B2B setting. Each one is a place the existing governance stack does not cover. Each one has an acceptable module candidate and a falsifiable prediction.
The audit layer is not another module in the governance stack. It is the counterparty that stack never had — standing next to the person who received the AI output.
Do toB first: the institution deploys, and is accountable to its own users. The first partner is Shielk, not because of content or marketing, but because of architecture — they already sit on the intercept between user and model, and they only use the outbound direction. The inbound direction is empty.
Behind every compliance claim sits a type of evidence. The market only produces the first two. The regulation's test is written on the third. The commercial logic of this brief sits in that gap.
Existence evidence · checklist
Is the component on the screen. Cheap, automatable, regression-testable. Proves "we put a disclosure there." Does not prove it did anything. Today's UI audit tools all stop here.Process evidence · documents
PIA / ÉFVP, ROPA, Model Card, FRIA. Proves the organisation thought about it, wrote it down, and someone signed. This is what platforms like Valdra produce, and the only thing B2B procurement currently accepts.Effectiveness evidence · live measure
Did the reader receive it, and did it change the judgment. The only evidence type that maps onto clear / informed / meaningful. Needs subjects, counterbalancing, and a behaviour log — no governance product is producing it.Where this layer sits: it does not compete with Type 1 or Type 2. It supplies the third kind, the only one that makes those two claims stand up. For a partner, this is an evidence upgrade of their existing line, not a replacement.
From the question going out to the evidence coming back, the audit layer has four places that must land. The first two are engineering. The third is the variable this POC is measuring. The fourth is the commercial close.
Inbound intercept — turn the proxy around
A de-identification proxy sits between the user and the model. Before send, it tells you which fields will be masked. When the answer comes back, nobody on that same pipe looks at it. It lands in front of the user as-is. The user has to decide whether this answer can be trusted, and is holding nothing.
Mount an audit hook on the existing proxy's return path. Do not touch the outbound path, do not touch the masking logic, do not add send latency — that is the first condition for convincing their engineering team. The audit runs in parallel as a stream after the response returns. The answer arrives first; the observation arrives later.
The token↔original map the proxy holds is the only basis the audit layer has for asking "did this answer use a masked key fact." The map never enters the audit module. The audit module only sends field-type questions ("did this response cite a POLICY-class field"); the proxy answers with a boolean.
The audit never blocks the answer. Observations arrive on a second stream. The UI reserves anchors but does not reserve empty space — if the audit times out, the interface must look as if the audit layer never existed, not leave a loading wreck.
- Audit latency slows the answer → users treat the whole pipe as "the thing that got slower." The first thing the institution kills is this layer.
- The audit module restores tokens to judge integrity → turns a privacy product into a privacy incident.
- The observation arrives too late → the user has already scrolled away, or already acted. An audit that speaks after the fact is an audit that did not speak.
Already paid
Four-eyes in finance and healthcare, cross-check in aviation — regulated industries already paid the cost of "a second pair of eyes on the same output."Beachhead
Institutions that already bought a de-identification proxy: they already admitted this pipe needs a middleman. Adding a direction is a much lower psychological bar than starting from zero.Enterprise trigger
The first enterprise security questionnaire that writes "did the user understand the limits of the AI output" into the procurement form.Gap generation — who writes Observation #3
The most important observation in the POC — "ACS family-history criteria were not checked" / "renal-injury risk was not raised" — was written by a person in advance. That is fully legitimate in a research setting: the gap is the experimental stimulus; it must be controllable. To go into B2B deployment, the same observation must be produced by the system at runtime, and it cannot be less reliable than the model it audits.
The auditor does not re-reason. It does one thing the model did not: walk the domain's authoritative list item by item. "Which guidelines / which risk dimensions should this decision consider" is a finite, maintainable set. "Did those items appear in the answer" is decidable. Reasoning stays with the model. Roll-call stays with the audit.
POC Observation #6 already named this type: lists ten sources, applies one. This is mechanically detectable — compare the source list against the citations that actually carry the conclusion. This type needs no domain knowledge. It is the first one that can land across industries.
If the audit layer is uncertain about a gap, it outputs "unchecked", not "no gap". Three states (gap / no gap / unchecked) must not be collapsed into two. "Unchecked" is useful information to the user. Folding it into "all clear" is this layer's most serious breach of trust.
- False positives accumulate → users learn to skip every observation. The audit layer becomes a second cookie banner.
- The list goes stale → the audit layer accuses a new answer with an old guideline, and manufactures error. Lists must be versioned and the version marked on the observation.
- Auditor and auditee share a source → the same model's blind spots appear on both ends. The layer is a dead letter that still emits an "already checked" signal.
Already paid
Clinical decision support and drug-interaction checks have run for decades — healthcare already paid the methodological cost of "list against output."Beachhead
Industries where the domain list already exists and has a maintainer: healthcare, tax, insurance underwriting. The list is the moat.Enterprise trigger
The first claim for "the AI assistant missed a material risk," and the underwriting questionnaires insurers write after it.Presentation — anchored vs sidebar
Once the observation is generated, where does it go. Format A anchors each observation next to the paragraph it points at (tooltip). Format B folds seven of them into an expandable panel (list). The information is identical — the difference is whether the reader will encounter it, and what they are reading when they do.
The observation is anchored to the content block it points at, with a badge and a connector. The bet: attention does not need to be transferred. The gap hits the user when they reach that paragraph. The risk is crowding and interruption, and users treating the badge as decorative footnote.
Seven observations sit in an "Audit Review" panel, expandable, readable as a batch. The bet: a complete audit opinion is more persuasive than fragments, and it does not interrupt reading of the answer itself. The risk is it competes with the answer for attention, and the answer always wins.
SEQ table, stepper, persona mechanism, and logging are all content-independent and correct. Only the content of slots 1–4 needs replacing. The four tasks must be mutually exclusive in clinical decision type (not rewrites of the same decision), and each must carry a gap type that M2.1 / M2.2 can detect.
- Neither format shows a significant effect → this is a publishable and commercially valuable result: it falsifies the assumption that "disclosure exists = compliant," which is why this layer exists. Do not treat it as failure.
- Data collection starts before learning effects are cleared → within-subject comparison collapses into between-subject. Sample-size demand explodes.
- Observations written too well → effect comes from copy quality, not position, confounding the independent variable. Copy across both formats must be identical, word for word.
Already paid
Mode annunciation in aviation, alert grading in clinic — safety-critical industries already paid the design cost of "the state must be seen."Beachhead
Product teams already running usability tests: they have a participant pipeline and a process. They lack this specific dependent variable.Enterprise trigger
The first supervisory guidance or procurement clause that asks for an "effectiveness test report" instead of a "disclosure screenshot."Effectiveness evidence — write back to the risk register
This is the most fragile link in the brief, and it has to be faced: a component that tells the user, at runtime, "this answer missed a material risk" looks, at first glance, like it increases the institution's exposure. If this objection cannot be answered, the first three insertion points — however well built — will not sell.
Output format aligns with the governance platform's existing mitigation fields: control name, mapped risk, deployment scope, efficacy metric, review cycle. Wording is always "this institution deployed X to mitigate Y," never "this institution's interface failed to achieve Z." Same number. Two frames.
Efficacy metrics are population statistics: reach rate, open rate, median dwell, stratified by format. No field that can be traced to an individual user — this is both a privacy requirement and a commercial one: institutions will not buy something that logs their users' every move.
Every efficacy record binds to the compliance claim it supports (which document, which section, which adjective). That is the first time the evidence chain closes: the document says informed → this is the measurement of informed → this is the method and the date.
- Read as audit, not control → counsel kills it on first contact. Wording and output fields must be control vocabulary from day one.
- Ugly numbers get suppressed → selective disclosure, once discovered, destroys the layer's credibility — worse than an ugly number. The output contract must include a "regardless of result" commitment.
- Becomes a rubber stamp for self-certification → if the institution can tune the metric, this layer immediately degrades to Type 1 evidence. The measurement method must be fixed externally.
Already paid
Post-market surveillance for drugs, clinical evaluation for devices — regulated industries already paid the institutional cost of "you still have to evidence effectiveness after deployment."Beachhead
Institutions already being chased by enterprise procurement questionnaires, who already bought a governance platform: they are already paying to produce Type 2 evidence, and they cannot answer the effectiveness question.Enterprise trigger
The first supervisory guidance that lists effectiveness measurement as a condition for recognising a mitigation; or the first fine on the grounds that "disclosure was not sufficient to constitute notice."Five swimlanes: principal, audit agent, de-identification proxy, model, product UI. Red is the existing outbound path (do not touch). Blue is the inbound path this layer adds.
Invariants · must not be violated
- The answer is never delayed by the audit. If the audit fails, the UI reverts to looking like there is no audit layer.
- The original-value map never enters the audit module. The audit asks type, never content.
- When uncertain, output "unchecked." Never output "no gap."
- Efficacy records are aggregate only, never individual. Output regardless of whether the result looks good.
| # | Phase | Actor | Data boundary | Failure handling | Gradient |
|---|---|---|---|---|---|
| 1 | Question sent | User → Proxy | Original text stays on the user side and the proxy. The audit module is not in yet. | Proxy offline → institution policy decides whether to pass; the audit layer does not join that decision. | Already paid |
| 2 | Outbound de-identification | Proxy | PII replaced with tokens; the map lives only on the proxy. Existing capability. Do not change it. | Mask fails → existing fail-closed logic. This layer does not intervene. | Already paid |
| 3 | Response returns | LLM → Proxy | Full answer + source list enter the proxy. | Model timeout → existing error handling; the audit layer produces no observation. | Already paid |
| 4 | Gap roll-call | Audit Agent | Gets the full answer and the domain list; does not get the map; queries the proxy with type questions only. | List version stale → output "unchecked" and mark the list version. Do not guess. | Beachhead |
| 5 | Answer first | Proxy → HMI | The answer does not wait for the audit. The UI renders in full first. | Audit timeout → no leftover UI elements, as if there is no audit layer. | Enterprise trigger |
| 6 | Observation anchored | Audit Agent → HMI | Observations arrive on a second stream, with region anchors and a three-state flag. | Anchor dead (content changed) → degrade to panel. Do not misplace a badge. | Enterprise trigger |
| 7 | User reads | User | Open / skip events logged locally; no Q&A content. | All skipped → that session is zero reach, counted in the efficacy denominator. Do not retry or nag. | Enterprise trigger |
| 8 | Efficacy aggregate | Audit Agent | Aggregate stats, minimum group 20; bound to a concrete compliance claim. | Sample too small → do not emit a metric. Do not pad with a low-confidence number. | Beachhead |
| 9 | Write back to register | Audit Agent → Governance | Output in mitigation-field format; attach method and date. | Field mapping fails → emit standard JSON for manual attach. Do not silently drop. | Beachhead |
Three brands look like three relationships. They are one relationship, three doors — the content front is not a standalone SaaS. The sort key is architectural fit: who owns the position this layer needs.
- Whether the three brands are in fact three fronts of one organisation — that decides "one relationship, three doors" vs three separate negotiations.
- Whether the de-identification proxy is a synchronous in-line proxy — that decides whether I1 is an integration (weeks) or a rebuild (months).
- The actual text and current guidance of AI Act Art. 14 / Art. 50, Law 25 s.12.1, and PIPEDA consent — the adjectives this brief hangs on (clear / informed / meaningful) are load-bearing. Check them against the current statute and guidance, and have counsel confirm whether they constitute an effectiveness standard.
- Until clinical sign-off and the task-set redesign are done, no efficacy number exists. Outbound proposals may cite method, not expected results.
The structured content of this document is embedded as application/json, id #ral-proposal. Any agent can read it without parsing the DOM. The schema reuses the existing pattern library's spec.rows contract (phase / actor / boundary / failure / grad), so this document can share the same retrieval and render pipeline as the Agent-based Onboarding series.
Suggested MCP tool contract below. Each insertion point maps to one tool. ral.audit_response is the only one that needs a runtime deploy; the other three are offline or aggregate, and can be built first to lower the technical bar of the first collaboration.
// I2 + I3 — runtime: generate observations and pick the presentation ral.audit_response( answer_text: string, // full model output sources: string[], // sources the response claims to use domain: "clinical"|"tax"|…, // which authority list to apply checklist_version: string, // list version, written onto the observation format: "anchored"|"panel" ) → { observations: Observation[], checklist_version, coverage: "checked"|"partial"|"unchecked" } // Observation — same contract as the POC static array; drop-in replacement Observation = { id, region, side, label, title, text, gap: boolean, state: "gap"|"clear"|"unchecked" } // I1 — boundary query: ask type, never take the original value ral.field_type_query(response_id, field_class: "POLICY"|"NAME"|…) → { used: boolean } // I4 — aggregate efficacy, min group 20, no user-identifying fields ral.efficacy_record(session_events[], bound_claim) → { n, format, gap_reach_rate, gap_open_rate, median_dwell_ms, bound_claim: { document, section, adjective }, method, measured_at } // Retrieve: module and criterion library; shared schema with Agent-based Onboarding ral.list_modules(gate?: "I1"|"I2"|"I3"|"I4") → { modules: Module[] }