Gemma GRC: A Technical Specification

This is the reference document for Gemma GRC: what each part does, how data moves between the parts, and the evidence behind each design choice. The story of how it got here is in It Learned How I Sound, Not What I Know, The Law Moved in July and Before I Fixed the Model, I Had to Fix the Grader. The run-by-run lab record is in When the Instrument Is the Bottleneck. This document is versioned, and when a number changes, the version changes with it.

At a glance

Part	Summary
What it is	A question-answering assistant for GRC work that answers only from a governed library and cites every claim
Where it runs	A Mac Mini M4 with 24 GB of memory, with no network connection at question time
Generator	Gemma 4 E4B (Q4_K_M) on Ollama, thinking off, greedy decoding, no fine-tuned adapter; it replaced Gemma 3 4B on 13 October
Library	5,989 chunks from 58 official public sources in six lanes, plus 315 chunks of my own writing and teaching
Retrieval	BM25 and BGE-small embeddings fused by reciprocal rank, top eight, with a router for questions about when rules apply
Agent access	One owner-only MCP tool that returns text for public and own passages and counts for everything else
Evaluation	749 questions: 457 answerable, 284 whose answer is not in the library, 8 canaries
Judges	Rules, then FactCG claim by claim, both tested on 272 controls before use
Baseline	Gemma 3 4B (Run 011): 82% of claims supported by a right passage, against 23% from memory; 57% of questions the library cannot answer declined
Shipping	Gemma 4 E4B (Run 013b): 95% of claims supported by a right passage; 93% of those questions declined; 95% of cited claims supported by the passage they cite
Status	Working system; every result is provisional until the test set is frozen

Design principles

1. Read, don’t remember. Facts live in the library, where they can be updated in an evening and checked in a minute. The model supplies language and discipline, not facts. When I fine-tuned a small model on my notes, fabricated details rose from one answer in five to nearly half.
2. Show me the page. Every factual claim cites a passage the model was shown. When the library does not cover a question, the answer says what is missing and where it would be.
3. Govern what goes in. A fail-closed gate decides what enters the library. Every passage records where it came from and what its license allows, and its provenance class decides where its text may go.
4. Label everything. Every regulatory passage carries its citation, version date, application date and status: in force, draft or superseded. Superseded texts stay in the library, labeled, because “what did this replace?” is a common question.
5. Test the instruments. Retrievers, judges and the test set are measured before anything is measured with them.
6. The model is replaceable. Because the knowledge lives in the library, swapping the generator is a configuration change followed by a rerun of the tests. Runs 013 and 013b did exactly that.
7. Local by default. Answers are produced on the Mac. Text leaves the machine only when its provenance class allows it.

System overview

The system has three stages. Build turns sources into a labeled, fingerprinted index, and no model is involved. Serve answers a question from that index with no network connection. Measure scores answers against a frozen test set, using judges that have themselves been tested.

The library

Public lanes

Primary texts come in six lanes, one for each regulatory area I work in. For every source, the manifest records the official URL, jurisdiction, instrument, document type, status, version date and license, and whether the text may be used to train a model.

Lane	Scope	Sources	Chunks
1	The EU AI Act as amended by the AI Omnibus (Regulation (EU) 2026/1744), the Commission’s guidelines and codes of practice, the AI literacy Q&A, and three draft high-risk classification guidelines	14	1,647
2	UK critical third parties: the 2026 designation regulations, FSMA 2000 Part 18 Chapter 3C, HM Treasury’s approach and announcement, the Bank of England, PRA and FCA rules and statements, and the memorandum of understanding between the ESAs and the UK authorities	10	457
3	DORA and the oversight of critical ICT third-party providers: the Regulation, seven delegated and implementing acts, and the ESAs’ designation list and work programme	10	535
4	Model risk: SR 26-2 and the 2026 interagency guidance, OCC Bulletin 2026-13, FDIC FIL-15-2026, the superseded SR 11-7 and SR 21-8, and the PRA’s SS1/23	7	162
5	EU cyber resilience: the Cyber Resilience Act and the Commission’s guidance on it, NIS2 and its implementing regulation, ENISA’s technical guidance and single reporting platform	6	902
6	NIST: SP 800-53, CSF 2.0, SP 800-171r3, the SSDF, the AI RMF, AI 600-1, AI 100-2 E2025, and four drafts	11	2,286
	Total	58	5,989

Forty-nine sources are in force, seven are drafts and two are superseded. Twenty-seven are under CC BY 4.0, seventeen are US federal works in the public domain, five are under the UK Open Government Licence (one of them with the FCA Handbook terms), and four are ESA publications reusable with acknowledgment. Five UK regulator texts are licensed for internal use only. They can be retrieved on the Mac, but they never leave it and never train anything.

Fetching

The fetcher accepts only the official hosts of the issuing bodies: the EU institutions and agencies, the UK government, legislation.gov.uk and the UK regulators, the US banking agencies, and NIST, including NIST’s own GitHub organization for the OSCAL catalogs. It follows redirects itself, forcing https and checking the host again on every hop, and refuses anything else.

EUR-Lex answers scripts with a bot challenge: an HTTP 202 and an empty body. The fetcher never tries to get past it. EU acts come instead from the Publications Office’s CELLAR repository, requested by CELEX number with content negotiation for XHTML.

Every raw file is hashed with sha256 and logged with its retrieval date, so any passage can be traced to the exact bytes it came from.

Parsing and labels

Format	Sources	Unit
EUR-Lex XHTML	13 EU acts, original and consolidated	Article, recital or annex, with consolidation markers removed
UK CLML XML	2 instruments from legislation.gov.uk	Regulation or section
OSCAL JSON	4 NIST catalogs	Control or enhancement
CPRT JSON	The NIST AI RMF	Subcategory
PDF	31 guidance documents, statements and rules	Page, with running headers and front matter removed
HTML	7 letters, bulletins and web pages	The document body

Units are packed into chunks of at most 1,700 characters of body text. Each chunk starts with a header that carries its citation and labels, so the label travels with the text into the prompt:

SR 11-7 guidance p. 8: III. OVERVIEW OF MODEL RISK MANAGEMENT (part 3 of 3)
SR 11-7 / OCC Bulletin 2011-12 attachment | version 2011-04-04 | [SUPERSEDED by SR 26-2 (17 Apr 2026); kept for history]

Drafts carry [DRAFT: not adopted; do not state it as law]. Each chunk also stores structured fields: a content-hash ID, provenance, jurisdiction, instrument, status, version date, application date, license, whether it may train a model, official URL, retrieval date, document, unit and formal citation.

My notes

Notes and course material from my vault pass the gate described on the project page. A note is held back if it has a classification, sensitivity or handling label other than public, a classification line at the top, a private or confidential flag or tag, or a handling banner. Quirks in line endings or frontmatter cannot open the gate. Cleared notes are chunked, PII is redacted, and each chunk is stamped with a content-hash ID and a provenance class: own, for my own writing and teaching, or a class that stays local.

Two builds

The provenance class decides which build a chunk joins.

• The clean build holds public texts and my own writing: 6,304 chunks. It is the only build used for evaluation, examples and anything published, and it is indexed with the stock BGE-small model.
• The local build adds every other class the gate clears, for 7,167 chunks. It serves my own questions on the Mac, with the search model I fine-tuned in August. That model was trained before the gate existed, so it never leaves the machine.

Each index is stamped with fingerprints of its corpus and its embedding model, and it refuses to load against anything else. Chunk IDs are content hashes, so a passage keeps its ID across rebuilds unless its text changes.

Answering a question

Hybrid retrieval

The question is ranked two ways: BM25 over the chunk text, and cosine similarity over BGE-small embeddings, with BGE’s query instruction added to the front of the question. The two rankings are merged by reciprocal rank fusion with the usual constant of 60, and the top eight passages are kept. There is no reranker; one made results worse when I tried it.

Temporal routing

Questions about when a rule applies were the weakest category. Asked when the Annex III obligations of the AI Act apply, the search ranked the Act’s descriptive articles highly and never surfaced Article 113, the article that sets the dates. The model then supplied a stale date and cited passages that contained none.

The router is a narrow fix for that failure. When a question asks when something applies, enters into force or takes effect, the router works out which instrument it means: by name, through aliases in the manifest, or else as the document that dominates the top 20 results. It then puts up to two of that instrument’s application provisions first, meaning the articles headed entry into force, application, commencement, transposition or transitional provisions. Everything else keeps its fused order.

Question set	Right passage in the top eight, before	After
Canaries	6 of 8	8 of 8
Temporal probes	6 of 10	10 of 10
Eval v3 temporal questions	13 of 13	13 of 13

The router can be switched off, and Run 011’s router-off arm does exactly that.

Context and prompt

Passages go into the prompt in rank order, each clipped to 2,000 characters, until a 13,000-character budget is used up. Long passages fill the budget early. Across Run 011’s 450 questions, 115 prompts held six or seven passages rather than eight. The prompt is short, and its four rules do most of the work:

Rules:
- Cite the context number you used, like [2], for each factual claim.
- If the context does not contain the answer, say exactly what is missing and which document would hold it. Do not guess.
- Never invent article numbers, control IDs, dates, or URLs.
- If a passage is marked [DRAFT], [PROPOSAL] or [SUPERSEDED], say so whenever you rely on it; never present it as current law.

Generation

Gemma 4 E4B runs on Ollama as a Q4_K_M build, with greedy decoding and at most 500 new tokens. It has about four billion effective parameters, eight billion in all, and an Apache 2.0 license. Ollama turns thinking on by default for Gemma 4, so every call turns it off. It replaced Gemma 3 4B instruct, a 4-bit build on Apple’s MLX, on 13 October, after Runs 013 and 013b (see The generator). The model carries no fine-tuned adapter. On an earlier setup, retrieval raised answer quality by 0.86 points on a five-point scale (95% CI 0.48 to 1.24). Fine-tuning on top of retrieval added 0.04 (CI -0.34 to 0.44), which is nothing. After each answer, the tool lists its sources with citation, status, version date and official URL.

One answer, end to end

A real run from 28 September, on the clean build, with Gemma 3 4B, the generator at the time. The question was: “Our model risk policy still cites SR 11-7. Is that still current, and does the current guidance cover generative AI?”

The search returned eight passages: six pages of SR 11-7, each labeled superseded; one of my lecture notes; and page 3 of the 2026 interagency guidance, which says generative and agentic AI models are not within its scope. The answer said SR 11-7 was superseded by SR 26-2 on 17 April 2026, and cited the label. Its fourth sentence then said the new guidance does cover generative AI, and its fifth said it does not.

FactCG scored each cited sentence against its passage: 0.80, 0.88, 0.20 for the misread, 0.95 and 0.81. The judge caught the one claim a reader should not trust. Asked the same question with no library, the same model said SR 11-7 was still current and dated it to 2019. Gemma 4 E4B, which replaced it, read the generative AI scope right in all six samples of the matching canary in Runs 013 and 013b.

Agent access

My agents reach the library through one MCP tool, search_grc, served over stdio on the Mac. It is registered for my own sessions only. My Discord agents get it only on turns I start myself, never on a handoff from one agent to another, and the grant lives outside every repository.

The tool returns passage text only for public texts and my own writing. Matches in any other class come back as counts, so an agent knows something relevant exists and can tell me to look locally. Public passages arrive with their citation, status, version date, official URL and a handling note derived from their license.

This is the one path where library text reaches a model off the machine. The agents run on a hosted model, so only text whose class allows it to leave is ever returned. Answers from ask.py never leave the Mac.

Measuring it

Eval v3

Eval v3 is the test set every change is measured against. It is built from the clean build only, so everything about it can be discussed in public.

• Questions. qwen2.5 14B wrote questions from sampled units. A deterministic check dropped any question that did not name its instrument or authority, and any about URLs, contact details or page numbers. Of 883 drafted, 749 are usable: 457 answerable, 284 absent (the library does not hold the answer; 134 more were dropped because it did) and 8 canaries. The absent set was tripled on 12 October, before the freeze, so that a 6-point change in refusals could be measured (see Runs 012 and 012b).

• Canaries. Eight hand-written questions about facts that changed in 2026:

  • the AI Act’s new application dates;
  • the new wording of the AI literacy duty in Article 4;
  • when the Commission’s fines for general-purpose AI providers became enforceable;
  • the four UK critical third parties;
  • SR 11-7’s replacement, and what that replacement leaves out of scope;
  • the Cyber Resilience Act reporting duty that went live on 11 September.

  A system answering from stale memory fails them, so every run includes them, whatever the split.

• Holdout. Questions are split by source unit, not by question, so no unit feeds both dev and test. That gives 919 units, 153 dev questions and 596 test questions.

• Several right answers. Each question gets a set of gold passages, not just one. Candidates are pooled from four retrievers (BM25, BGE, BGE-M3 and the hybrid) to depth 6, and two local judges grade them for relevance from 0 to 3. Gemma 4 E4B grades every pair; qwen2.5 14B grades the pairs Gemma 4 rated as at all relevant, plus a random 15% of the rest.

• Grading the graders. Claude also grades a pair when both the question and the passage are public: 1,581 pairs, for evaluation only, and never as training data. When the local judges disagreed, Claude sided with qwen2.5 14B 85% of the time, so qwen settles disagreements on the pairs that never leave the Mac.

• My audit. I grade a blind sample of 100 pairs, drawn from Claude’s grades, the tie-breaks and the jury, with no other grade on screen. When the audit is done, the set is frozen and its sha256 recorded.

Judges

Answers are scored claim by claim, on the Mac, by judges from model families other than the generator’s.

• Tier 0: rules. No model is involved. An answer is split into claims, and openers, sign-offs and headings such as “Here’s the answer to your question, based solely on the provided context:” are not claims. Every citation must point at a passage that reached the prompt. Every date, number and article, section or control reference in a claim must appear in the passages it cites, and dates match in any format. Abstentions, including the hedged (“the context does not provide”) and closed-book (“I don’t have information”) kinds, are scored as abstentions.
• Tier 1: FactCG. FactCG-DeBERTa-v3-Large (0.4 billion parameters, MIT license) scores whether a passage supports a claim. It needs its authors’ instruction template; called as a plain sentence pair, it cannot tell right from wrong. Each claim is checked against its cited passages first, then against every shown passage if those do not support it, then against the gold passages. A score of 0.5 or more counts as support.
• Tier 2: candidates. Granite Guardian 4.1 8B (groundedness) and gpt-oss-20b (holistic, on binary criteria) have been tested but are not in use yet.

Before a judge is used, it runs on 272 controls built from the test set, where the right verdict is known:

Control	Count	Right verdict
Positive: the reference answer, correctly cited	51	Pass
Quote positive: a verbatim sentence from the gold passage	23	Pass
Swapped: the reference answer with a date or number changed	6	Flag
Quote swapped: the verbatim sentence with a detail changed	19	Flag
Added: the reference answer plus an unsupported claim	51	Flag
Wrong citation: the right answer citing the wrong passage	51	Flag
Wrong abstention: “not in the passages” when it is	51	Flag
Wrong answer: a confident answer when the library has none	20	Flag

FactCG gets 269 of 272 right, so it is the claim judge. The rules catch changed details, wrong abstentions and wrong answers, but not added claims or wrong citations. That is expected: they are a cheap first filter.

The two LLM judges failed most quote positives, and the fault is partly in the controls. The quote is the first sentence of the gold passage that carries a date or number, whether or not it answers the question. FactCG ignores the question and passes them, while the LLM judges read the question and call them non-answers. Until the quote controls are rebuilt, Tier 2 stays unused. Granite Guardian also needs its thinking mode on, because without it the model calls everything ungrounded, and it takes about 20 seconds per claim.

The relevance judges were tested the same way, against Claude’s grades on 300 pairs. Gemma 4 E4B agreed on 48% (Cohen’s kappa 0.14), and qwen2.5 14B on 82% (kappa 0.50). Gemma 4 E4B is being replaced as a juror, and now that it is also the generator there is one more reason. Its part in the labels is narrow. Disagreements go to Claude or to qwen, never to Gemma 4. The pairs only Gemma 4 graded are ones it ruled out, and in the 15% sample qwen never found one of those relevant (0 of 532).

One rule sits above all of this: only open-weight judges from families other than the generator’s may drive a training decision. Claude’s grades evaluate; they never train.

Metrics

A claim is a sentence or list item of an answer, after openers, sign-offs and headings are dropped.

Metric	Definition
Correct claims	Share of claims that a gold passage supports
Grounded claims	Share of claims that a passage the model was shown supports
Citation precision	Share of cited claims whose cited passage supports them
Fabrication	Answers with a claim supported by no gold or shown passage, or with a date, number or reference found in none of them
Cites unseen	Answers with a bracketed number that is not a passage the model was shown
Wrong abstention	Answerable questions the model declined although a gold passage was among those shown
Declined, unshown	Answerable questions the model declined when no gold passage was shown; reported, never counted as wrong
Gold hit	A gold passage among those shown
Absent abstention	Questions with no answer in the library that the model correctly declined
Canary pass	The answer states the changed fact (a date in any format), does not state the stale one, and does not abstain

Samples are averaged within a question first, then across questions, with 95% bootstrap intervals. Two runs are compared question by question with a paired bootstrap.

Until scorer 2026-10-13.1, wrong abstention counted every declined answerable question, including those whose gold passage never reached the prompt. Declining there is what the prompt asks for. The new definition was written down before Run 013b, for the reason given there.

Run 011

Run 011 is the first end-to-end baseline, of the system as it shipped until 13 October. It runs on the eval v3 test split, 589 questions after the 12 October extension, plus all 8 canaries:

• Router on: the shipping configuration at the time, three samples per question at temperature 0.7.
• Router off: the same, with the temporal router disabled. Where the router does not change the passages, the prompt is identical, so the answer and its score are reused.
• Closed book: the bare question with no library, one sample, as in the August baselines.

It answers two questions: how much the library is worth compared with the model’s memory, and whether the router helps or hurts outside date questions. The eval is a draft until my audit is done, so the results are provisional; freezing it means scoring again, not generating new answers. Router off reused 1,758 of its 1,791 answers; only the 33 answers to the 11 questions whose passages the router changed were generated again.

Run 011 results

Scored with scorer 2026-10-13.1, on the eval v3 test split, which has 227 questions the library cannot answer since the 12 October extension. The scorer changed only how wrong refusals are counted, and no figure below moved except the no-library arm’s: it is never shown a passage, so its refusals no longer count as wrong.

Paired by question, with 95% intervals:

Comparison	Measure	Difference
Library against memory	Claims supported by a right passage	+58.9 points (55.8 to 62.0)
Library against memory	Answers with an unsupported claim	-72.7 points (-76.5 to -68.5)
Library against memory	Refuses when the library has no answer	+53.6 points (48.6 to 59.2)
Library against memory	Canaries passed	+83.3 points (66.7 to 100)
Router on against off	Every answerable measure	Within 0.3 points; every interval includes zero
Router on against off	Canaries passed	+20.8 points (4.2 to 45.8)

The library carries the facts. The same model, asked the bare question with no library, supports 23% of its claims with a right passage and almost never declines. With the library, it supports 82% and declines just over half of the questions it should.

The router does what it was built for and nothing else. It moves the date canaries and leaves every other measure where it was, so it stays on.

“Unsupported” is mostly one sentence. 276 of 1,086 router-on answers have a claim the judge could not tie to a passage, and 212 of them have exactly one. That claim is usually commentary, an inference, or a sentence about the answer itself. Invented dates, numbers or references are rare: 7 claims in 1,086 answers.

Answering what the library cannot was the weakest behavior. 43% of questions whose answer is not in the library still got an answer, assembled from nearby passages. Runs 013 and 013b fixed most of it.

Citations are looser than claims. 63% of cited claims are supported by the passage they cite, though 90% of claims are supported by some passage the model was shown. Of the 90 answers with a bracketed number that is not a shown passage, 83 carry numbers above 8, which are reference markers copied from the source text, such as a NIST document’s [154]; 7 point at a passage the context budget cut.

Canaries: 20 of 24. Two of three samples said the Commission’s fines for general-purpose AI providers apply from 2 February 2025, not 2 August 2026, and two of three said the 2026 model risk guidance covers generative AI.

Measuring the measurement

The first scoring of Run 011 had faults of its own, found by reading the answers rather than the averages:

• Openers were judged as claims. Answers that open with “Here’s the answer to your question, based solely on the provided context:” were marked fabricated 63% of the time, against 17% without one.
• Canary checks were literal. “July 13, 2026” failed a check for “13 July 2026”, italics in “not within the scope” failed a correct answer, and an answer that began “Yes, the guidance applies to generative AI” passed because a later sentence quoted the right phrase.
• The refusal detector was narrow. It missed “the context does not provide”, and its closed-book form, “I don’t have information”, entirely.

Scorer 2026-10-01.1 fixes all three, and every arm was scored again without generating a single new answer:

Router on	First scoring (2026-09-29.2)	Rescored (2026-10-01.1)
Answers with an unsupported claim	36.0%	25.4%
Canaries passed	70.8%	83.3%
Refuses when the library has no answer (the original 80 questions)	44.2%	56.2%
Claims supported by a right passage	79.2%	81.6%

The rescored canary results match a line-by-line read of all 24 answers by Claude, the evaluation-only grader. The first scores are archived beside the new ones, never mixed with them.

Other faults surfaced while the run was live:

• Before any answer was scored. The scorer treated a date as unsupported when the passage wrote it in another format, read stray punctuation into amounts, counted a detail that sat in a different shown passage as fabricated rather than as a citation error, and accepted citations to passages the context budget had kept out of the prompt.
• While generating. The run stopped at answer 1,057 with a Metal out-of-memory error.
• While scoring. The scorer’s GPU cache grew to 27 GB, and a bracketed date, “[2026-07-27]”, crashed the citation parser.

All were fixed, and two answers generated again after the memory fixes matched the originals word for word.

Prompt experiments: Runs 012 and 012b

Run 011 left two targets: questions the library cannot answer, and loose citations. Both were tried through the grounding prompt. v1 stays the shipping default, and each candidate sits behind a gate: a quick pass on the dev split stops the run if more than 5% of answerable questions are refused.

• Run 012, prompt v2. It added three rules at once:

  • a scope check;
  • an opening line to use when the passages do not answer (“The context does not contain the answer.”);
  • a rule to cite only passages that state the claim.

  The gate stopped it after 22 minutes, with 96.8% wrong refusals. The model opened 92 of 95 answerable answers with the quoted sentence, and 90 of those went on to answer anyway: a 4B model treats a quoted opening line as mandatory. The citation rule made no measurable difference.

• Run 012b, prompt v3. v1 plus the scope check alone, with no quoted sentence. It passed the gate, and the full test run was compared with Run 011 question by question.

Run 012b against Run 011 (paired, 95% interval)	Difference
Refuses when the library has no answer (227 questions)	+3.5 points (-1.3 to +8.5)
Wrongly refuses an answerable question	+1.3 points (+0.2 to +2.7)
Claims supported by a right passage	+1.9 points (-0.4 to +4.3)
Answers with an unsupported claim	-4.3 points (-8.0 to -0.6)

The decision rule, written before the run, adopts a change only if refusals rise with an interval above zero.

• On the first 80 no-answer questions the gain was 6.3 points, with an interval from -1.2 to +14.2: too wide to call. Rather than loosen the rule, the no-answer questions were tripled through the same pipeline.
• On 227 questions the gain is 3.5 points, still not distinguishable from zero.

v3 is not adopted, and v1 keeps shipping. Its clearest effect, fewer unsupported answers, came with more wrong refusals and would need a run of its own.

The generator: Runs 013 and 013b

Two prompt runs left the refusal problem where it was, so the next change was the model. Run 013 swapped Gemma 3 4B for Gemma 4 E4B and changed nothing else: the same clean build, prompt v1, eight passages, the router, and three samples at temperature 0.7, with the sampler settings matched across runtimes. The runtime changed with the model, from MLX to Ollama, because that is how it would ship. The decision rule, written before the run, would adopt Gemma 4 only if:

• claims supported by a right passage fell by no more than 2 points;
• answers with an unsupported claim, and wrong refusals, each rose by no more than 2 points;
• refusals on no-answer questions, supported claims or citation precision improved, with an interval above zero;
• and an answer took no more than twice as long.

Run 013 failed one criterion. It passed the dev gate, with 2.1% wrong refusals and no empty answers, then improved every quality measure by a wide margin. But wrong refusals rose 2.7 points (0.8 to 4.7), over the limit, so by the letter of the rule Gemma 4 was not adopted.

Reading the refusals explained it. Gemma 3’s few refusals were mostly answers that ended by saying the context does not specify some detail. Gemma 4’s were real refusals. On 7 test questions, retrieval never put a gold passage in the prompt. Gemma 3 answered all 21 samples anyway; Gemma 4 declined 16 of them. Declining when the answer is not in front of it is what the prompt asks for, but the metric counted it as wrong. That split was made after seeing the result, so it could not decide Run 013.

The fix was written down first, then tested on fresh samples. Scorer 2026-10-13.1 counts a refusal as wrong only when a gold passage was shown, and reports the rest on their own line. Stored scores were converted rather than judged again, and every other figure for every run stayed the same. Run 013b then repeated Run 013 with new sampling seeds and nothing else changed, and the decision rested on Run 013b alone.

Gemma 4 E4B passed every criterion and replaced Gemma 3 4B on 13 October. Run 013b agrees with Run 013 to within one point on every measure, and the canaries match question by question.

• Refusals. It declines 93% of the questions the library cannot answer, against 57%, while wrongly refusing 2% of answerable ones.
• Grounding. 95% of its claims are supported by a right passage, and 6% of its answers carry an unsupported claim, against 25%.
• Citations. 95% of cited claims are supported by the passage they cite, against 63%. It has almost stopped copying source reference markers such as [154].
• Speed. About twice as fast, on the same Mac.
• Canaries: 19 of 24, against 20 of 24. It reads the generative AI scope of the 2026 guidance right in every sample, where Gemma 3 got it wrong in two of three. But in two of three samples it says SR 11-7 is superseded without naming SR 26-2, and in none does it give 2 August 2026 as the date the Commission’s fines on general-purpose AI providers start.

One check came after the run. Gemma 4 E4B also graded relevance for eval v3, so it had a hand in deciding which questions count as unanswerable. On the 52 no-answer test questions where a second judge confirmed every relevance label, it declines 94%, against Gemma 3’s 71% (+23.1 points, 11.5 to 34.0), close to its 92% on the rest. Its own vote does not explain the gain.

Findings so far

Finding	Evidence	Consequence
Fine-tuning a small model on my notes taught it my style, not my facts	Fabricated details rose from 20% to 48% on a 1B model and from 26% to 36% on a 4B model	The generator ships without an adapter
Retrieval is the largest gain	Plus 0.86 on a five-point scale against closed book (95% CI 0.48 to 1.24)	The library became the product
Fine-tuning the search model paid off where fine-tuning the generator did not	The right passage in the top four rose from 71% to 79% after nine minutes of training	Tune the failing component, not the famous one
The instruments failed before the model did	A judge that scored 4.99 of 5 on everything, a tokenizer call that hid a NaN, a missing query prefix, and a noise floor of about 0.3 points at 50 questions	Every instrument is control-tested, and the test set has 749 questions, not 50
Date questions failed on retrieval, not reading	Article 113 was not in the top 60 results for “when” questions	The temporal router took the canaries from 6 of 8 to 8 of 8
The generator misreads	Gemma 3 4B said the 2026 guidance “does cover generative AI” against a passage saying it does not, and FactCG scored the claim 0.20; in Run 011, two of three samples of the scope canary said the same	Claim-level judging decided the generator swap; Gemma 4 E4B reads that scope right in six of six samples
The library carries the facts	Run 011: claims supported by a right passage, 82% against 23% from memory (+58.9 points, 95% CI 55.8 to 62.0)	Every later change is measured against this baseline
The router helps date questions and nothing else	Canaries +20.8 points (4.2 to 45.8); every answerable measure within 0.3 points	It stays on
Answering what the library cannot was the weakest behavior	With Gemma 3 4B, 43% of the 227 questions with no answer in the library got one anyway; two prompt changes did not move it (Run 012b: +3.5 points, -1.3 to +8.5)	The generator was tried next
A small model copies a quoted line	Prompt v2’s quoted refusal opened 92 of 95 answerable answers, and 90 of them answered anyway	Never give the 4B model a sentence to open with; change one rule at a time
The scorer needed the same scrutiny as the model	Openers counted as claims, literal canary checks and a narrow refusal detector moved the headline numbers by up to 12 points	Every score carries its scorer version; old scores are archived, not mixed
The generator was the lever	Gemma 4 E4B against Gemma 3 4B (Run 013b): refusals +35.7 points (29.7 to 41.3), unsupported answers -19.4, citation precision +31.8	Gemma 4 E4B ships from 13 October
A grounded model exposes retrieval misses	On 7 questions with no gold passage in the prompt, Gemma 3 answered all 21 samples and Gemma 4 declined 17 (Run 013b)	A refusal counts as wrong only when a gold passage was shown, a rule written before Run 013b; retrieval is the next lever
A rule can change, but only before the run	Run 013 missed one criterion because of how refusals were counted; the fix was written down first and tested on a replication	Decision rules are fixed before a run and judged on a run that has not been seen

What it is for

Today

• “Where exactly does it say that, and is it still current?” A cited answer in seconds, with the version and status of each source.
• Regimes side by side. Incident reporting under DORA and NIS2, or oversight of critical technology providers in the EU and the UK, with both sides cited.
• Questions whose answer moved. The AI Act timetable, SR 26-2, the UK designations: maintained text beats memorized text.
• Finding what is missing. When it cannot find something, that usually marks knowledge that lives in someone’s head. In a compliance function, that is key person risk.
• A research tool for agents. My research, content and teaching agents cite primary text through the MCP tool instead of their own recall.

Not for

• Settling an ambiguous interpretation, predicting how a supervisor will read an answer, or standing in for legal advice. That judgment stays with the professional.
• Confidential or client material on a personal machine. The regulator-response work below belongs in an organization’s own environment.

Next capabilities

Policy alignment. Break a regulation into obligations: who must do what, by when. Match each obligation to the policy clauses that should cover it. Classify each as covered, partially covered, missing or in conflict, with both texts cited side by side. For each gap, draft a revision tied to the obligation it closes.

The draft can be automated; the change cannot, because a policy edit needs an owner, an approval and change control. The error that matters most is a false “covered”, so the first metric is precision on “covered” against a hand-labeled set of obligations and clauses.

Regulator response memory. Treat every response sent to a supervisor as a guiding document, with its context attached: which regulator, when, the question, the scope, the commitments made and the facts relied on. When a similar question arrives from a different regulator, the assistant would:

1. Find what was said before.
2. Check that the scope really matches.
3. Flag facts that have changed since.
4. Flag anything that would contradict what another regulator was told.
5. Draft, with sources.

Supervisors share information about the providers they both oversee. The ESAs and the UK authorities signed a memorandum of understanding on critical technology providers in January 2026, and it is in lane 2.

This capability does not belong on a personal computer. Responses to supervisors can be confidential supervisory information, which the supervisor controls, and legal should sign off before any of it is indexed. My project stays on public texts and my own notes, and it proves the methods: what goes into the library, how each passage is labeled, and how each capability is tested.

What comes next

Each item has a definition of done.

1. Freeze eval v3. My 100-pair audit comes first; if my agreement with Claude is low, the answer key gets rebuilt. Done when the sha256 is recorded and Runs 011 and 013b are scored again against the frozen set.
2. Judges. Rebuild the quote controls from the sentence that answers the question, rerun Granite Guardian and gpt-oss, and decide whether a Tier 2 judge earns its cost. Replace Gemma 4 E4B as a relevance juror with a model from another family, now that it is the generator. Done when the control table above is rerun with every judge.
3. Retrieval, now the main lever. A gold passage misses the prompt for 1.9% of test questions, and with Gemma 4 those become refusals rather than unsupported answers. Add stemming to BM25, trial granite-embedding-english-r2 as the dense model, retrain the search model on clean data only, and add canaries for phrasings that miss. “Do our generative AI models fall under SR 11-7?” currently retrieves no page of SR 26-2. Done when gold hit improves on the frozen set without a canary regressing.
4. Naming what replaced what. Two canaries got worse with Gemma 4: it calls SR 11-7 superseded without naming SR 26-2, and it misses the date the Commission’s fines start. Test a prompt rule to name the replacing instrument and its date, on its own, against v1 on Gemma 4. The citation rule planned earlier is dropped, since 95% of cited claims are already supported by the passage they cite.
5. Policy alignment prototype. On public texts and a sample policy, measuring false “covered” first.
6. Serving. Offer retrieval as a local tool to the small model my other agents use for local work, with the provenance boundary enforced at the tool.

Known limitations

• It still answers some questions the library cannot. Gemma 4 E4B declines 93% of the 227 questions whose answer is not in the library; the other 7% get an answer assembled from nearby passages. Gemma 3 4B declined 57%.
• It declines when retrieval misses. When no right passage reaches the prompt, which happens for 1.9% of test questions, it says the library does not cover the question. That is the intended behavior, but the library does hold the answer.
• The generator misreads names and dates. Across Runs 013 and 013b, Gemma 4 E4B called SR 11-7 superseded without naming SR 26-2 in four of six samples, and never gave the date the Commission’s fines for general-purpose AI providers start. Gemma 3 4B misread negation and list scope instead: it wrote that the 2026 guidance covers generative AI one sentence before saying it does not.
• Eight passages is really six to eight. The 13,000-character budget cut the prompt to six or seven passages for 115 of 450 questions.
• Phrasing matters. BM25 has no stemming, and a question that names an old instrument can pull back only that instrument’s pages.
• It over-answers dates. Asked about Annex III, the model lists every date in Article 113.
• The generator is also a relevance juror. Gemma 4 E4B graded relevance for eval v3 before it became the generator. It never settles a disagreement, and the refusal gain holds where a second judge confirmed every label, but the overlap ends only when it is replaced as a juror.
• The numbers are provisional. The eval is a draft until the audit, and every result records the hash of the eval it used and the version of the scorer that judged it.
• One machine, one user. There is no availability, access control or change management beyond what a personal lab needs, and none of this is legal advice.

Appendix: stack and reproducibility

Models

Software

Python 3.12. Ollama 0.34.4 serves the generator and the local judges; since version 0.34, a re-pulled Gemma 4 thinks by default, so every call turns thinking off. Evaluation runs set Ollama’s sampler to MLX’s settings (top-p 0.95, no top-k), so a model swap does not also change the sampler. MLX 0.32.3, with mlx-lm pinned to a commit on its main branch for the CVE-2026-5843 fix, runs the Gemma 3 4B baseline. FactCG runs on PyTorch with Apple’s MPS backend. MLX’s buffer cache is capped at 1 GiB and emptied after every answer, and PyTorch’s MPS cache is emptied after every FactCG call; without both, Run 011 hit a Metal out-of-memory error and the scorer grew to 27 GB.

Reproducibility controls

• Chunk IDs are content hashes, and the index refuses to load against a different corpus or embedding model.
• Every raw source file is hashed at fetch time, with its retrieval date.
• The question-answering tool decodes greedily. Evaluation runs pin a seed for every question and sample, derived from a seed base that a replication changes.
• Every run writes its configuration (model, build, k, router, temperature, seed base, eval hash, corpus fingerprint, prompt hash, and library versions, including Ollama’s version and the model’s digest) and refuses to resume under a different one.
• Every score records the version of the scorer that produced it and the verdict for each claim. Scores from another version are archived, never mixed, and scores are reused across runs only within a version. A version that only redefines a measure converts stored scores from fields they already hold, instead of judging again.
• A test set may grow under a run only when every question the run has already answered is still in it, and the replaced eval fingerprints stay in the run’s configuration.
• 94 unit tests cover the rules, parsers, router and scoring without loading a model.

Versions of this document

• v1.3. Gemma 4 E4B replaces Gemma 3 4B as the generator after Runs 013 and 013b; scorer 2026-10-13.1 counts a refusal as wrong only when a gold passage was shown, and Run 011’s refusal figures are restated under it; findings, next steps, limitations and figures 1, 4 and 9 updated, as of 13 October 2026.
• v1.2. The no-answer questions tripled (80 to 227 in test; 749 usable in all), Run 011 refusal figures updated, the prompt experiments (Runs 012 and 012b) and their verdict, and the next steps reordered, as of 12 October 2026.
• v1.1. Run 011 results under scorer 2026-10-01.1, how the scorer changed and why, and refreshed findings, next steps and limitations, as of 2 October 2026.
• v1.0. The system as it stood on 29 September 2026, before the Run 011 results.