Every non-obvious choice in this project, with the reasoning behind it. The failed experiments stay in. So do the diagnoses that turned out wrong, and the features that were built, measured, and then dropped. A log holding only successes is a sales document.
Every entry is written as its own decision is made or its experiment runs, not reconstructed afterwards. Reverted experiments stay in. A change killed by its own number is the evidence that this log is real rather than a narrative written backwards from a finished system.
Entries are DL-n. Handbook defect identifiers are D-n and live in corpus/handbook/DEFECTS.md. They are not the same series.
Status: open, resolves in Phase 5
Hypothesis. A legal-domain embedding model beats a general-purpose one on retrieval recall for statutory text, because the corpus is dense with terms of art (FMLA, 825.200, serious health condition) that carry precise meaning.
Metric. recall@10 across the full scenario set, and separately on the conflict slice, where the correct chunk is often not the most semantically similar one.
Why it is not decided up front. It is a plausible hypothesis, not a known fact. Both models get embedded and measured before either is adopted.
Result. pending
Status: decided, Phase 1
Decision. The handbook, the seeded defects and roughly 90 scenarios are authored before any retrieval, embedding or agent code is written.
Why. Ground truth cannot be generated by the system it is meant to evaluate without becoming circular. If scenarios were drafted after the pipeline worked, they would be shaped, consciously or not, by what the pipeline already does well. The eval would then measure self-consistency rather than correctness.
Second reason, equally practical. The scenario set is what tells ingestion when to stop. Without it there is no definition of "enough corpus," and state-law ingestion expands indefinitely because there is always another statute that feels relevant.
Cost accepted. Phase 1 produces nothing runnable. There is no demo, no metric and no visible progress until Phase 5. That is the trade.
Status: decided, Phase 1. Verification pending Phase 3.
Decision. Every statutory claim in DEFECTS.md and every required_citation in a scenario carries an unchecked verification box until it has been confirmed against the actual ingested text. No scenario is scored while a box it depends on is unchecked.
Why. The defects file was drafted from recall of California, New York and federal leave law. Recall is a plausible starting point and an unacceptable ground truth. If the eligibility threshold written here is wrong, then every metric computed against it is wrong in a way no amount of downstream rigour can detect, and the whole project quietly measures nothing.
Consequence. Where verification contradicts the drafted ground truth, the ground truth changes. The statute is not adjusted to fit the test.
Status: decided, Phase 1
Decision. The scenario set is balanced across seven slices with counts committed in advance: straightforward 15, ambiguous 15, control 10, conflict 20, superseded 10, out of scope 10, adversarial 10. Ninety total.
Why. The headline comparison in this project is agent versus naive baseline. Conflict cases are exactly where the baseline fails and the agent wins, so a set weighted toward conflicts would inflate the delta without any engineering behind it. Committing the balance before any number is visible removes the temptation to discover, later, that a more favourable mix was the more "representative" one.
Enforced in code. tests/test_scenario_balance.py asserts the exact counts, so drifting the mix breaks a test rather than quietly improving a metric. The targets are written into the test file with the date they were committed.
Status: decided, Phase 1
Decision. For each behaviour the agent can get right by being cautious, the set contains a paired case where that same caution is wrong.
clarify is wrong.D-1) is paired with handbook-above-statute (D-2), so "statute always wins" is wrong.D-5) is paired with a personal leave question the ambiguity does not affect.Why. Without pairing, an agent that always asks a clarifying question and always defers to statute scores well while being useless. Over-clarification is the failure mode users actually experience, and it is invisible unless the eval contains cases where asking is the wrong move.
This is the metric that most distinguishes the agent from the baseline, and also the one most likely to make the agent look worse before it looks better.
Status: decided 2026-08-26. Keep New York; scope the adapter to what the scenarios need.
Observation. With all 90 scenarios written, the jurisdiction distribution is Ohio 39, California 29, withheld 16, New York 6.
Why it happened. The contrast that generates interesting cases is "state exceeds the federal floor" versus "state adds nothing." California and Ohio sit at the two extremes and produce clean pairs. New York is a third point that mostly differs in mechanism, an insurance-based benefit rather than a broader entitlement, which is genuinely interesting but harder to write pairs against.
The tension. New York is the most expensive layer to ingest: it is the only source needing an API key, its own adapter, and a tree traversal, and it currently supports 6 scenarios. Ohio needed the least work and supports 39.
Options.
Leaning toward 1, because the mechanism difference is real: an eligibility test based on weeks worked rather than months-plus-hours is exactly the kind of thing a system trained on federal shapes gets wrong. But that is an argument for a handful of targeted cases, not for parity.
Keep New York. Do not write more scenarios. Ingest only the Article 9 provisions the existing seven touch.
The reasoning changed while building Phase 2. The value of New York was never scenario coverage. It is that it is a third integration shape. Federal is a structured XML API with a separate versioning feed. California and Ohio are server-rendered HTML that must be parsed. New York is an authenticated JSON API with a ragged tree traversed by depth. For a project whose subject is wiring a model into whatever heterogeneous systems a customer already runs, that variety is the substance rather than a detail. Three sources of one shape would demonstrate less than three sources of three shapes.
Seven scenarios do not justify ingesting the whole Workers' Compensation Law. They comfortably justify the Paid Family Leave provisions they actually cite, which is also what the plan already requires: ingestion stops when the scenario set is answerable, not when the corpus feels complete.
Writing scenarios to justify a corpus would be backwards. Cases manufactured to make an adapter look worthwhile are worse than thin coverage, because they inflate the set without testing anything a real user would ask. If NY ingestion proves cheap, cases can be added opportunistically; that is not a commitment.
Accepted cost. The New York layer stays the most thinly tested of the three. Any claim about jurisdiction handling should say so rather than implying even coverage.
Status: decided and executed, Phase 1.5
What happened. After the 90-scenario set was written and merged, a reviewer with no stake in the design audited it against the spec. It found defects severe enough that several metrics would have been uninterpretable, including three that were missing rules rather than bad data.
Why it was not caught by the author. DL-2 says ground truth cannot be generated by the system it evaluates without becoming circular. The same circularity applies one level up: the person who wrote the scenarios knows what each one meant, so ambiguity in what it says is invisible to them. Every defect below was obvious to a first-time reader and none were visible from the inside.
Scope of the review, deliberately bounded. The reviewer was explicitly forbidden from assessing whether the statutory claims are legally accurate, because it shares the author's blind spots there and would have agreed confidently. That verification remains deferred to Phase 3 against ingested text (DL-3). A model reviewing a model's legal recall produces agreement, not evidence.
Three missing rules, not bad data.
No tie-break for expected_authority. Precedence rules 1 to 4 only resolved layers that disagree. Where a handbook restated a statute, the label was arbitrary, and the set applied it inconsistently: the same restatement pattern was labelled federal in one place and company in another. Precedence correctness was unscoreable. Spec rule 5 now says the controlling authority is the highest-authority source that independently compels the outcome.
refuse and escalate were not separable. The definition of refuse was "nothing covers this, here is who to ask", which is also a description of escalate. Eighteen scenarios rode on a line that had never been drawn. The boundary is now subject matter: refuse when nothing in the corpus bears on the question, escalate when it does but a correct response needs human judgment.
Route accuracy rewarded never clarifying. With 57 answer and 15 clarify scenarios, a system that never asks a clarifying question had an 83% micro-accuracy ceiling while scoring zero on the behaviour DL-5 exists to test. Worse, test_clarify_is_outnumbered_by_answer actively enforced that imbalance. DL-5 guarded over-clarification and nothing guarded the reverse. Route accuracy is now macro-averaged.
A consequence that changed the schema. Rule 5 exposed a case nobody had considered: the answer can be determinate while the controlling authority is not. Sick leave accruing at one hour per thirty worked is the same number in every state, but California compels it by statute and Ohio only through the handbook. Demanding one authority would penalise a correct system for a fact it was never given. That is what acceptable_authorities exists for.
Data defects. A scenario declared a fact missing while supplying it. A pairing was asserted in prose and was simply false, the two scenarios being unrelated. Another control case duplicated a conflict case and withheld nothing. Four scenarios in the conflict slice contained no conflict, one of them a duplicate. Two defects the scenarios depended on were undocumented (now D-7, D-8). One control case turned on a conditional catch-all, making a clarifying question defensible and the ground truth wrong.
Tests that did not test what they claimed. test_unverified_scenarios_are_tracked asserted every scenario was unverified, so it would have failed the moment Phase 3 verified its first one: a test that breaks when the project succeeds. The supersession test checked only that forbidden_citations was non-empty, not that it named the other version. No test resolved handbook citations against the handbook. No test checked pairings at all, which is how the false pairing survived.
Prose intent is not enforcement. Every property the design depended on that lived only in a notes field had drifted. Every property encoded as a field with a validator had not. So pairs_with, must_address, acceptable_authorities and weeks_worked_12mo are now fields, and the loader checks that pairings resolve and are reciprocal, and that every handbook citation names a policy that exists.
Writing the new pairing test immediately caught a defect in itself: it initially treated two clarify scenarios as contrasting on nothing, when they legitimately contrast on which prong is missing.
DL-4 fixes the slice balance in advance so it cannot be tuned to flatter results. Two counts changed here, both in the direction of a harder set:
A balance change that makes the score harder to achieve is the opposite of the failure DL-4 guards against. Both are recorded here so the direction is auditable.
Roughly a day, before anything depended on the ground truth. The same defects surfacing in Phase 5 would have meant every metric in the project was measuring something other than what it claimed, with no way to detect it from inside the system.
Status: decided and implemented, Phase 2
The trap. The eCFR point-in-time endpoint returns the text of a part as it stood on a given day. The obvious implementation reads that snapshot and stamps every section with the date requested. It is also wrong in a way that is invisible until it matters.
A snapshot answers "what did this say on 1 August 2026". It cannot answer "since when", because the text returned may have been unchanged since 2016. Stamping the snapshot date as effective_from would make every federal provision appear to have begun on the day we happened to fetch it, and any query asking what the rule was in 2020 would find nothing in force.
The fix. A second endpoint, /versioner/v1/versions/title-29.json, carries per-section amendment history. For Part 825 it reports 132 version rows across 79 sections with 16 distinct amendment dates. effective_from is the latest amendment at or before the snapshot; effective_to is the day before the earliest amendment after it, or None where none exists. The snapshot date is retained separately as observed_on.
Corrected in Phase 2.5, and the correction is the interesting part. This entry originally described the result as "real regulatory history". Review checked the feed and found that all 79 sections carry a shared 2016-12-01 row, whose issue_date is the eCFR baseline load. For 74 of 79 sections that date records when the text entered the electronic record, not when the provision was promulgated: Part 825's own SOURCE note gives 78 FR 8902, February 6 2013.
So the fix for DL-8 had quietly reproduced DL-8's own error one level down, trading "the date we fetched it" for "the date the database first held it" and calling both an amendment date.
effective_from_is_floor now marks these, baseline_load_date() detects the shared date structurally rather than hardcoding it, and the SOURCE and CITA provenance the parser previously discarded is captured into source_note. Queries before the floor return nothing in force, which is the honest answer: we cannot attest to the text then. No scenario is affected, since the earliest as_of_date in the set is 2023.
Refusing to guess. A section with no amendment record raises rather than defaulting. A fabricated date would silently corrupt every point-in-time answer touching that section, and nothing downstream could detect it.
Reserved sections are kept, not dropped. Three sections in Part 825 are [Reserved]. They carry no text and are marked rather than filtered, so a gap in the numbering reads as a fact about the corpus rather than a parser failure. Same principle as the Ohio absence records.
Status: federal layer verified, Phase 2. State layers still pending.
All nine federal citations used in the scenario set resolve against the ingested corpus, and each load-bearing claim was checked against the actual text rather than accepted from recall:
| Claim | Source | Result |
|---|---|---|
| 12 workweeks of leave | 29 CFR 825.200 |
Confirmed |
| Eligibility: 12 months and 1,250 hours | 29 CFR 825.110 |
Confirmed, wording is "at least", which settles the inclusive boundary in conflict-017 |
| Covered employer: 50 or more employees | 29 CFR 825.104 |
Confirmed |
| 30 days notice where foreseeable | 29 CFR 825.302 |
Confirmed |
| Group health coverage maintained | 29 CFR 825.209 |
Confirmed |
| Same or equivalent position on return | 29 CFR 825.214 |
Confirmed |
| Qualifying reasons enumerated | 29 CFR 825.112 |
Confirmed |
| Certification may be required | 29 CFR 825.305 |
Confirmed |
One claim was wrong as written. DEFECTS.md D-4 stated that federal FMLA "does not cover grandparents". Verification shows 29 CFR 825.122 does mention grandparents, but only inside the definition of next of kin of a covered servicemember, where they sit in the priority order for military caregiver leave. Ordinary care leave for a sick grandparent is genuinely uncovered, so conflict-005 survives, but the blanket claim did not.
This is exactly the failure mode DL-3 was written to catch: a statement that is nearly right, drafted from recall, that would have produced a defensible-looking metric while resting on an overbroad premise. It cost one line to fix here. Left in place, a scenario about a wounded servicemember grandparent would have been scored backwards with no way to notice.
12 of 92 scenarios are now marked verified. The rest depend on state law or on the Ohio absence records and stay unverified until Phase 3. Nothing is scored while its dependencies are unchecked.
Status: decided and executed, Phase 2.5
A second independent review, this time of Phase 2. Two findings changed how the project is built rather than what it contains.
The reviewer mutated end_dates_from_versions in three ways: flipped the day offset from minus to plus, removed the offset entirely, and loosened a comparison operator. The full suite passed every time. The tests asserted effective_to >= effective_from and "some values are non-null", both of which survive an arithmetic that is simply wrong.
The date arithmetic turned out to be correct, verified independently against the raw feed. That is not the point. The tests were not the reason it was correct, and would not have caught it becoming incorrect.
Tests now pin actual derived values (825.120 ends 2018-06-26; 825.300 ends 2018-01-01, the snapshot date itself; exactly two sections are superseded after a 2018-01-01 snapshot) and exercise in_force_on at both edges. All four mutations are now caught.
Adding a boundary case took two rounds. After the first rewrite, one mutation still survived: no snapshot in the suite fell exactly on an amendment date, so the inclusive comparison was untested. A snapshot of 2018-06-27, a real amendment date, closed it.
The general lesson. For derived values, assert the value. A relationship between two derived numbers is satisfied by an entire family of wrong derivations.
SourceDocument was designed against one well-structured federal source and had absorbed its shape:
source_id and citation held the same string for all 79 sections, leaving no stable key distinct from the display citation, and no way to key two versions of one handbook policy.version or supersedes link, which the superseded scenario slice depends on.is_reserved was a boolean expressing an eCFR concept, and could not represent an Ohio absence record, which is a different thing: a positive finding that a layer says nothing.section_path repeated the document's own heading, duplicating it into every chunk.<P> elements only, silently dropping anything held in another element.source_url pointed at /current/ even for point-in-time documents, so a provision that ceased in 2018 linked to today's wording.All are fixed. content_status is now a three-way enum where absent carries retrievable text, so the agent can find "no provision exists" rather than getting an empty result and being unable to distinguish it from a retrieval failure.
Fixing a contract after one adapter is cheap. After four it is a migration.
Adapters cache to a gitignored directory, so a stray call in a test would not fail: it would succeed, write a cache file, and every later run would pass off data nobody chose. conftest.py now blocks the socket layer outright.
All 12 scenarios marked verified: true in Phase 2 have been reverted. Every one rests on something unbuilt: the Ohio absence records, or California and New York text not yet ingested. control-004 claims an answer is "uniform across the corpus" while two thirds of the corpus does not exist yet.
DL-3 says a scenario is verified when all its dependencies are checked. Federal-only was not the same as fully verified, and the distinction was collapsed under the satisfaction of having verified something.
Status: Phase 3. Federal and California verified; New York and the Ohio absences outstanding.
Reading the citation requirements off the scenario set before writing any adapter:
| Jurisdiction | Citations required |
|---|---|
| California | 4 sections |
| New York | 1 section |
| Ohio | 0 |
Ohio needs no statutory adapter at all. All 48 Ohio scenarios resolve to federal law, the handbook, or a recorded absence.
Two corrections to this paragraph, made in Phase 3.5. The count was 46 and is 48; it was written from memory rather than computed. And "or a recorded absence" was not true when written: three scenarios rested on Ohio silence about jury duty pay, witness duty pay and military leave that no record covered. A test now asserts every Ohio scenario expecting the handbook to control has a corresponding absence record, so the claim is enforced rather than asserted. The plan's stopping rule, that ingestion ends when the eval is answerable, eliminated an entire integration before a line of it was written. The Phase 3 plan had specified ingesting Ohio's military, jury duty and voting leave provisions; nothing asks about them.
conflict-017 was backwards. Federal 29 CFR 825.110 grants eligibility at "at least 12 months". California Gov. Code 12945.2 says "more than 12 months of service". At exactly twelve months the federal test is met and the state test is not, so federal controls, not state. The scenario had been written assuming both floors read alike, with a note asserting the inclusive reading as though it settled both. It is now a better scenario than the original: a boundary case where the two layers genuinely diverge by a day of service.
Cal. Lab. Code 227.3 does not say what D-6 claimed. The defect asserted that a use-it-or-lose-it clause is unenforceable in California generally. The section addresses forfeiture and payout upon termination and is silent on annual carryover during employment. The wider proposition rests on vacation being vested plus agency interpretation, neither of which is in the corpus. conflict-008 and ambiguous-004 were reframed onto termination, which the text does support, and conflict-009 moved to Ohio.
This paragraph originally said conflict-007 was reframed too. It was not. The edit silently no-opped and nothing checked. See DL-12.
Both errors are the same shape: a claim that is true enough in general and not supported by the specific text cited. Neither would have failed a plausibility check. Both would have scored a correct system as wrong.
One near-miss worth recording: an early grep for CFRA's employer threshold searched for "five or more employees" and reported it missing. The statute says "five or more persons". Verification that greps for the phrasing you expect will report an absence that is not there.
Phase 2.5 reverted 12 scenarios marked verified: true that rested on unbuilt Ohio absence records. In Phase 3, with that correction freshly written into the log, 16 were marked verified and an audit found 8 resting on exactly the same unverified absences, plus 2 claiming uniformity across a corpus that is only fractionally ingested.
DL-3 was unambiguous. It had just been restated. Judgment failed anyway, in the same direction, because verifying something creates the feeling of having verified enough.
So it is no longer a matter of judgment. Two tests now enforce it: no Ohio scenario may be verified while the Ohio absence records carry verified_on: null, and no scenario withholding jurisdiction may be verified at all, since such an answer claims something about every state and only a fraction of state law exists in the corpus.
Six of 92 scenarios are verified, all California-plus-federal cases whose every dependency is checked. That number is lower than the 16 claimed an hour earlier, and it is the first one that is true.
Status: decided and executed, Phase 3.5
Third independent review. The finding that matters most is not a bug.
DL-11 stated that conflict-007 had been reframed from annual carryover onto termination, because Cal. Lab. Code 227.3 addresses only the latter. It had not been. The scripted edit used str.replace(), the target text no longer matched because a pairs_with line had been inserted into the block earlier in the same session, and str.replace() returns the string unchanged when it matches nothing. Nothing asserted otherwise.
So the scenario kept a question the cited section does not answer, kept a note asserting the exact proposition DL-11 had just retracted, and was marked verified: true on the strength of a correction that never ran. Its paired scenario's note, "identical question, Ohio employee", was false as well.
A silent no-op is worse than a crash in exactly the way this project keeps rediscovering. The suite stayed green, the commit message described the intended change, and the decision log recorded it as done. Every artifact agreed, and all of them were wrong.
Every scripted edit now goes through a helper that asserts the target was found. That rule caught a second no-op within minutes of being adopted.
Six must_address values were written as [\"LEAVE-002\"] by an f-string escaping bug, and loaded as the literal string \"LEAVE-002\". The DL-7 citation guard did not catch them, because it only inspected strings that already looked like handbook citations. A guard that ignores what it cannot recognise is not a guard. It now rejects any citation containing a quote or backslash outright, and had to be narrowed once when it flagged N.Y. Workers' Comp. Law 204, whose apostrophe is legitimate.
Prefix matching. "12945." is a prefix of "12945.2.", so parsing the 12945.2 page as section 12945 emitted the right citation attached to the wrong statute, silently. Both sections are in this corpus, so it was reachable in ordinary use. Now matched with a boundary.
Four mutation survivors, all in California and New York parsing: the section-number offset, the credit-line trailing anchor, the hierarchy append, and New York's title strip. Each is now pinned by a test asserting a value.
New York's heading regex over-stripped. [^.]*\. after the section number consumed the body when a title contained a period, and ate the 1. opening the first subdivision when there was no title. It now anchors on the title the API supplies.
Two guard tests could silently disable themselves. The Ohio guard read verified_on by substring-matching source_note, which a YAML value of the string "null" satisfies identically to a real null. verified_on is now a parsed field on a typed record, and a string "null" is rejected. The jurisdiction-withheld guard had no exit condition at all, making it the DL-7 anti-pattern: a test that must be deleted the moment the project succeeds. It now self-disables once every jurisdiction is covered.
Two of the verified six were overstated. conflict-017 claimed the state test fails at exactly twelve months; subdivision (r) of 12945.2 admits air-carrier crew at "12 months or more", so the claim holds for this employee and not for California generally. And conflict-001 named state as controlling when at 14 months both federal and state independently compel the same outcome, which precedence rule 5 never ordered. That gap is now written into the spec, and the scenario carries acceptable_authorities.
The absence sentinel leaked semantics. Absences were dated 1900-01-01 with effective_from_is_floor=True, overloading a flag documented as "the earliest date the source can attest to" and implying the FMLA governed in 1900. in_force_on now short-circuits on content_status, so the sentinel is never compared against a query date.
Contract drift between adapters. Federal documents carried a real section heading while California and New York repeated their citation, leaving a chunker with no context for state documents. New York had a real title and was discarding it into source_note. California genuinely has none, since its sections are untitled, and that is now documented rather than left looking like an oversight.
Status: Phase 4. DL-1 and the chunking comparison resolve in Phase 5.
Chunking is now a measured question, not an assumption. Structure-aware splitting and a fixed-size window are both implemented behind one interface. The baseline overlaps, because beating a straw man proves nothing about the strategy that wins. On the real corpus, structure produces 300 chunks averaging 1,186 characters and fixed produces 313 averaging 1,276. (299/1,190 before Phase 4.5 fixed the prose degeneration described in DL-15.) Which is better is a Phase 5 number.
Embedding likewise. Both DL-1 candidates are implemented, neither is a default, and get_provider() has no fallback, so a choice cannot be made by omission. Both pin a model version rather than a floating alias: an index built against a silently updated model is not comparable with the numbers that justified adopting it.
What could not stay open: what gets embedded. A bare subdivision such as "(b) The determination is made under the principles of the FLSA" is close to meaningless alone, and this corpus is largely made of them. Chunks therefore carry their heading and nearest hierarchy level into the embedding while the stored text stays clean. Without it an FMLA chunk and a CFRA chunk read almost identically, which would make the state-versus-federal distinction unretrievable regardless of which model wins.
A dedup check that looked equivalent and was not. The fixed-size chunker dropped its final window when that window appeared inside the previous one. On repetitive text any short tail is a substring of what precedes it, and statutory text repeats constantly, so real content vanished from the end of uniform documents. Redundancy is now decided by position.
Certificates. Assembling the corpus end to end failed on every HTTPS fetch with CERTIFICATE_VERIFY_FAILED. The project pins Python through uv, and that interpreter does not see the macOS trust store; earlier fetches had run under system Python and hid it. ingest/http.py now makes the trust store explicit, which also matters for anyone else cloning this.
Payload indexes are silently inert in local mode. Qdrant's in-memory client warns that payload indexes have no effect. The tests were therefore never exercising the production path, so indexing was run against the live container: 299 chunks, all six indexes created, no jurisdiction leaks. Same failure shape as the compose healthcheck that had never been executed.
Client and server versions were mismatched, 1.19 against 1.12.4, far enough apart for the client to warn. Both are now pinned in step, in pyproject.toml and docker-compose.yml, with a comment in each pointing at the other.
Status: written 2026-08-26, before any retrieval quality has been measured. Resolves in Phase 5.
This entry exists to be checked against a result that does not exist yet. A decision log in which every prediction turned out correct is a log written backwards, so the prediction, the mechanism, the falsifier and the tie-break rule are all fixed here in advance.
Structure-aware wins overall on recall@10, and the margin is concentrated rather than spread evenly.
The mechanism, corrected before the experiment ran.
This entry originally claimed that a fixed window would separate the two eligibility prongs of 29 CFR 825.110, leaving "(2) at least 1,250 hours" retrievable with nothing saying what it governs. That was checked and it does not happen. All four documents in the corpus containing "1,250 hours" keep both prongs inside a single fixed chunk: the prongs sit about 60 characters apart and the window is 1,500. The prediction as first written was unfalsifiable, because its mechanism could not occur.
The real mechanism is broader and much more common: 205 of 313 fixed chunks begin mid-sentence. A window that opens with "or been paid for 60 percent of the applicable monthly guarantee" has lost the subject it modifies, and its embedding carries a fragment rather than a proposition. Structure-aware chunks begin at a subdivision boundary by construction.
Correcting this before Phase 5 rather than after is the whole point of writing the prediction down. Had the original stood, a structure-aware win would have been credited to a cause that never operated.
Where it should not. The straightforward slice mostly asks about handbook policies, which are short, already close to one chunk under either strategy, and structurally simple. Expect no meaningful difference there, and treat a large one as a signal that something else is going on.
Chunk-length variance. Structure-aware inherits the corpus's lopsidedness: it produces chunks up to the 4,000 character cap, while fixed-size never exceeds 1,500. A long chunk dilutes its embedding, because one vector has to represent more distinct propositions. If structure loses overall, this is the most likely reason, and the check is whether its losses cluster on documents whose chunks run past roughly 2,500 characters.
A concrete number that would surprise me: structure-aware losing the conflict slice. That is the slice its entire rationale is about, and losing it would mean the fragment story is wrong rather than merely outweighed.
A second falsifier, added with the corrected mechanism. If mid-sentence starts do not hurt retrieval, the 205 fragments cost nothing and the two strategies should land within noise, which the tie-break already resolves toward fixed-size.
If the two are within 2 points of recall@10, adopt fixed-size. The threshold is a choice rather than a measurement, and it is set before seeing any result precisely so it cannot be adjusted afterwards to justify the more interesting answer.
The reasoning: structure-aware is the more complex implementation, carrying a subdivision regex, a continuation-grouping rule, a packing heuristic and a hard cap. Complexity has to be paid for in measured benefit, not in a plausible story about why it ought to help. If the numbers are level, the simpler thing wins.
Both strategies keep the heading and nearest hierarchy level in the embedded text. That was settled in DL-13 for a separate reason: without it a bare subdivision is close to meaningless and an FMLA chunk reads almost identically to a CFRA one. It applies to both arms, so it is a constant here rather than a variable.
Status: decided and executed, Phase 4.5
Fourth review, and a 63-mutant harness. Twenty-four survived.
The spec calls for hybrid dense-plus-sparse retrieval with named vectors, because statutory text is full of terms of art that carry exact meaning and a dense embedding places 825.200 next to its paraphrases. Phase 4 built a single unnamed dense vector and said nothing about it. Not deferred, not argued against: simply absent, while DL-13 discussed chunking and embedding as though retrieval design were settled.
Unnamed to named is a breaking change to a Qdrant collection. Left until after Phase 5, it would have meant rebuilding every index, including the paid ones. Found by a reviewer reading the spec against the code, which is the only way an omission gets found: nothing fails when a requirement is missing, because there is no code to fail.
Hybrid is now implemented. Dense and sparse are retrieved separately and fused with reciprocal rank fusion, with the jurisdiction and date filters applied to each prefetch rather than after fusion, so they stay hard constraints instead of a post-hoc trim that can silently return fewer results than asked for. Term frequencies are sent raw and IDF is computed by Qdrant across the collection, because computing it locally would mean recomputing on every corpus change and being silently stale whenever that was missed.
Ids came from abs(hash(chunk_id)) % 2**63. Python randomises string hashing per process, so the same chunk received a different id on every run. Re-indexing inserted duplicates instead of updating, and this was confirmed on the live server: a re-run in a fresh process left 12 points for 6 chunks, with nothing raising. Now uuid5 over a fixed namespace, pinned in a test.
The same reasoning applies to sparse dimension indices, which are a stable BLAKE2b hash for exactly this reason: a token that lands in a different dimension each run would never match anything indexed earlier.
DEFECTS.md must never be ingested. The allowlist was case-insensitive, so LEAVE-009-defects.md, LEAVE-001-DEFECTS.md and leave-000-defects.MD all satisfied it, and the test asserting the property only grepped documents that had already loaded, so it never presented a hostile filename.
There are now two independent mechanisms: a case-sensitive pattern with a slug denylist, and a content check that refuses any body reading as answer-key material whatever it is called. Two rather than one because the failure is silent. An agent retrieving the answer key would score well on precisely the scenarios built to catch it, and nothing in the result would look wrong.
The nine handbook policies are prose under Markdown headings with no (a) markers at all. Every block folded into one unit, target_chars became inert, and a 1,612 character policy emitted a single oversized chunk while the baseline correctly emitted two. The strategy comparison was meaningless for the entire company layer. It now falls back to paragraphs, which is the document's own structure one level down.
_pack's hard split had only an upper bound on chunk length, so a mutant that dropped half the text of every oversized unit survived. A conservation test now checks nothing is lost.SUBDIVISION regex was never tested. Loosening it to "anything starting with a bracket" survived.One mutant survives and should: break to continue in the window loop is equivalent, since once a window reaches the end of the text every later one does too. Chasing it would add a test that asserts nothing.
dimensions was accepted by the OpenAI provider and never sent, so a non-default value would have built a collection of one width and written vectors of another. The eCFR cache key embedded date.today(), so the default path fetched a snapshot nobody had cached and two runs on different days would have compared different text. The handbook stripped any leading blockquote rather than only the supersession banner. SDK dependencies had lower bounds but no upper ones, which contradicts the pinning discipline DL-1's reproducibility rests on.
Status: written 2026-08-26, before Phase 5 measures anything.
Two pieces of standard retrieval design were absent from the spec and the plan entirely. Not deferred, not argued against: never written down. This is the second time that has happened, after hybrid search in DL-15, and both were found by someone asking what the system does rather than by anything failing.
Committed. It belongs in the agent's triage node in Phase 6, and it is not optional, because two of the system's hard filters cannot be applied without it.
A real question is "my grandma is ill and I'm her main carer, can I take time off?" Three things have to happen before that reaches an index:
conflict-004 and conflict-005 exist to catch. When it cannot be established, the correct output is a clarifying question rather than an unfiltered search.as_of date, or the effective-date filter silently uses today and answers a question about 2023 with 2026 law.None of this is retrieval sophistication for its own sake. Each one feeds a filter that already exists and is currently fed by scenario metadata rather than by anything the system derives.
Not committed, and deliberately so. The corpus is 300 chunks. Reranking earns its keep when a candidate pool is large and noisy, and adding it before a baseline exists means never learning which problem it solved.
The measurement distinguishes two failures that look alike from outside:
The gap between recall@10 and recall@3 measures the second directly: passages the system found and then buried.
Pre-registered rule. If recall@10 - recall@3 exceeds 10 points, reranking has real headroom and gets built. At or below 10, it does not, and the effort goes to the agent instead.
The threshold is a choice rather than a measurement, and it is fixed here for the same reason as DL-14's tie-break: chosen afterwards, it would justify whichever answer was more appealing at the time.
One caveat that would override the rule. If recall@10 itself is poor, the rule does not apply, because ordering a set that lacks the answer improves nothing. Fix retrieval first.
Both are worth building and neither is what makes this project non-trivial. In this corpus the correct answer frequently contradicts the most semantically relevant document: the handbook is the closest match and, where it falls below a statutory floor, the wrong answer. Perfect retrieval returns it first and a perfect reranker keeps it there.
That is a reasoning problem over retrieved evidence, not a retrieval problem, and no amount of tuning upstream of the agent touches it.
Status: sparse-only arm complete, Phase 5. Hybrid arm blocked on credentials.
The scenario set against the live corpus, both chunking strategies, sparse-only retrieval. Sparse matching with server-side IDF needs no embedding provider, so this arm could run while DL-1 waits for keys. It is a real retrieval mode, not a degraded one, and it is reported as its own configuration rather than as the hybrid result.
These are oracle-filter numbers. Jurisdiction and as-of date come from scenario metadata rather than from the question, because query rewriting is not built (DL-16). This measures retrieval given perfect query understanding. Presenting it as end-to-end performance would flatter the system by exactly the cost of the missing component.
57 of 92 scenarios carry required_citations and are scoreable. The rest expect a clarification, refusal or escalation, which have no retrieval target.
| slice | n | structure | fixed |
|---|---|---|---|
| adversarial | 2 | 1.000 | 1.000 |
| conflict | 18 | 0.667 | 0.500 |
| control | 10 | 0.700 | 0.700 |
| straightforward | 17 | 0.765 | 0.765 |
| superseded | 10 | 1.000 | 1.000 |
| overall | 57 | 0.772 | 0.719 |
Structure-aware wins by 5.3 points, and the entire margin is in the conflict slice. Every other slice is tied to three decimal places.
The prediction was: structure wins overall, the margin is concentrated rather than spread, and the straightforward slice shows no meaningful difference. All three hold. Straightforward is not merely close, it is identical at 0.765.
The tie-break fixed in advance was 2 points. At 5.3 it does not apply, so structure-aware is adopted, provisionally pending the hybrid arm.
This is stronger evidence than the overall number suggests, because the mechanism was corrected in Phase 4.5 before the run. The original claim, that fixed windows split eligibility prongs, was checked and found not to occur. The replacement claim, that 205 of 313 fixed chunks begin mid-sentence and carry fragments rather than propositions, is the one that predicted a conflict-slice-only margin, and that is what appeared.
Inspecting the verified scenarios individually, conflict-001, conflict-007 and conflict-021 all rank the handbook first, on precisely the scenarios where the handbook is the wrong answer. The spec's opening claim, that the right answer often contradicts the most semantically relevant document, is now visible in retrieval output rather than asserted.
It also explains why the six verified scenarios score far worse than the rest: they are entirely conflict-slice, the hardest cases by construction. Comparing that subset against the others measures difficulty, not correctness, which is why the table above is by slice.
recall@10 - recall@3 for structure-aware is 15.8 points, against the threshold of 10 fixed in DL-16. On the conflict slice specifically, structure has lower recall@3 than fixed (0.278 against 0.333) while having higher recall@10, which is the exact signature of retrieving the right document and burying it.
Held rather than acted on, for one reason: DL-16 says the rule does not apply when recall@10 is itself poor, and 0.772 under sparse-only is not the number this decision should rest on.
conflict-017 asks about "twelve months"; the statute says "12 months". A sparse tokenizer cannot bridge that and a dense embedding can. Cases like it are why the sparse-only arm understates the conflict slice, which is precisely the slice the chunking margin lives in.
DL-1 remains open and needs OPENAI_API_KEY and VOYAGE_API_KEY. Until the hybrid arm runs, both the chunking adoption and the reranking decision stand as provisional.
Status: decided, Phase 5. Four configurations, 57 scoreable scenarios, oracle filters.
| config | recall@10 | recall@3 | mrr | forbidden | headroom |
|---|---|---|---|---|---|
voyage-law-2 / structure |
0.895 | 0.825 | 0.723 | 0.035 | 7.0 pts |
voyage-law-2 / fixed |
0.877 | 0.781 | 0.691 | 0.053 | 9.6 pts |
voyage-2 / structure |
0.877 | 0.702 | 0.673 | 0.035 | 17.5 pts |
voyage-2 / fixed |
0.860 | 0.675 | 0.652 | 0.035 | 18.4 pts |
voyage-law-2The legal-domain model beats the general one by +1.8 points of recall@10 in both chunking arms, identical to a decimal place, which is itself reassuring: the effect is independent of the other variable.
The interesting number is recall@3: +12.3 and +10.5 points. Both models find the right document about equally often. The legal model is far better at putting it near the top.
That is what domain specialisation should buy in a corpus like this one. Every chunk is employment law, so "relevant to the question" is nearly free and almost everything qualifies. The hard part is discriminating between near-neighbours: knowing that a CFRA eligibility clause beats a general FMLA definition for a Californian at fourteen months. Separating signal from noise is easy here; ranking plausible against correct is the whole problem.
The comparison is clean because both arms are the same generation at the same 1024 dimensions from the same provider. Comparing against OpenAI, as originally planned, would have confounded domain specialisation with vendor.
Structure-aware wins by +1.8 points, in both model arms. The tie-break fixed in advance was 2 points. So fixed-size is adopted.
Under sparse-only retrieval the gap was 5.3 points and structure won outright. Under dense retrieval it is 1.8. The mechanism explains the shrinkage: DL-14 predicted structure would win because 205 of 313 fixed chunks begin mid-sentence and carry fragments rather than propositions. Lexical matching punishes that; a dense embedding largely does not care, because it encodes meaning rather than token boundaries.
At n=57, 1.8 points is one scenario. The tie-break exists precisely for that: one scenario of difference does not justify a subdivision regex, a continuation-grouping rule, a packing heuristic, a hard cap, and a prose fallback.
An honest admission about the rule. It was written against recall@10 alone, and structure also wins on recall@3, on mrr, and on forbidden-citation rate under the adopted model (0.035 against 0.053). Forbidden citations are a correctness failure rather than a ranking one, and the rule did not consider them. That is an under-specification I should have caught when writing it.
It does not change the outcome. Every one of those gaps is also worth about one scenario at this sample size, so the rule's underlying logic, that the evidence is too thin to buy complexity, holds across all four metrics rather than only the one it named.
Recorded cost: the adopted configuration scores 0.877 rather than the best-observed 0.895. Adopting the winner instead would mean deciding, after seeing data, that a threshold set in advance was wrong, which is the exact failure pre-registration exists to prevent.
The rule was to build it only above 10 points of recall@10 - recall@3 headroom.
voyage-law-2: 7.0 and 9.6 points. Below.voyage-2: 17.5 and 18.4 points. Above.Choosing the right embedding model removed the headroom a reranker would have chased. On the general model the rule fires clearly and a reranker would have been built, tuned, and credited with recovering buried answers, when the same recovery was available for free by picking a domain-appropriate model.
The adopted configuration sits at 9.6 points, near enough the threshold to revisit if the corpus grows. It is not near enough to override a rule fixed in advance.
Best configuration, by slice:
| slice | n | recall@10 | recall@3 |
|---|---|---|---|
| adversarial | 2 | 1.000 | 1.000 |
| control | 10 | 1.000 | 0.900 |
| superseded | 10 | 1.000 | 1.000 |
| straightforward | 17 | 0.941 | 0.941 |
| conflict | 18 | 0.722 | 0.556 |
Four slices are at or near perfect. Conflict is twenty points below everything else, and it is the slice where the correct answer contradicts the most semantically relevant document. Retrieval is not failing there so much as being asked the wrong question: the handbook genuinely is the closest match, and no embedding model will rank a statute above it on similarity alone.
That gap is the argument for Phase 6. It is a reasoning problem over retrieved evidence, and it is what the precedence rules exist to solve.
These are oracle-filter numbers: jurisdiction and as-of date come from scenario metadata, not from the question, because query rewriting is not built (DL-16). End-to-end performance will be lower by whatever that component costs.
Only 6 of 57 scoreable scenarios are verified, and all six are conflict-slice. Verification since this run began has found five ground-truth errors (three Ohio absence records, one New York misattribution, one flaw in the search method itself). Those corrections will change some scenarios, and this table will need re-running afterwards. The embedding cache makes that nearly free.
Status: applied, Phase 5.5. Corpus 101 to 104 documents. Ohio absences 4 of 8 verified.
Verification ran while the Phase 5 evaluation was in flight, deliberately read-only so ground truth could not shift under a measurement. It found five errors. None would have failed a plausibility check, and all five had survived at least one full independent review.
1. parental_leave claimed Ohio has no parental leave entitlement. Ohio Administrative Code Rule 4112-5-05(G) makes it unlawful sex discrimination to terminate an employee disabled by pregnancy where policy leaves insufficient maternity leave available. Pregnancy leave has a state source in Ohio; bonding leave does not. The record now distinguishes them.
2. vacation_forfeiture claimed Ohio does not classify vacation as wages. ORC 4113.15(D)(2) defines "fringe benefits" to include vacation pay, and (C) makes an employer a trustee of funds an agreement requires. Ohio enforces the policy's terms rather than ignoring vacation, which is a sharper reason for conflict-008 than the one originally given, and the exact inverse of California voiding the term itself.
3. military_leave claimed Ohio has no military leave statute. ORC 5903.02(B) grants USERRA-equivalent reinstatement rights under Ohio law, naming no employer category. This is not an absence: it is a state rule that deliberately tracks federal law. "No Ohio rule exists" and "Ohio's rule restates the federal one" reach the same answer for opposite reasons.
4. Two scenarios cited N.Y. Workers' Comp. Law 204 for a weeks-worked eligibility test that is not in it. 204 sets benefit amounts and duration and defines no eligibility. The rule is in 203, "Employees eligible for benefits under section two hundred four". The claim about New York was correct throughout; the citation named the provision the rule points to rather than the rule.
5. My search method was structurally wrong. Step one enumerated Title 41, Labor and Industry, on the reasoning that a private-employer leave obligation would live there. All three contrary Ohio provisions were outside it: one in the Administrative Code, one in Title 23 (Courts, for jury duty), one in Title 59 (Soldiers and Sailors). A single-title sweep would have cleared every one of them confidently.
Errors 1 to 3 are claims about absent law. Error 4 is a claim about present law aimed at the wrong provision. They arrive at the same conclusion from opposite directions: nothing automated catches either.
Throughout, every check passed. The adapters parsed correctly. The loader confirmed citations resolved. Retrieval returned the cited sections. The tests were green. A corpus can be faithfully, verifiably, and completely wrong, and the only thing that catches it is reading the source against the claim.
That is also why the scoping rule needed qualifying. DL-11 credited "ingest what the scenarios cite" with eliminating an entire Ohio adapter, and it remains right. But it inherits the accuracy of the citations it reads.
Three sections were added because a claim needed text that was not there: Cal. Lab. Code 246, Cal. Gov. Code 12945.7, N.Y. Workers' Comp. Law 203.
Two claims then verified cleanly. California bereavement is five days and unpaid, so the handbook's ten paid days genuinely exceeds it. WCL 203 confirms the twenty-six-week test.
The third exposed a further problem. Cal. Lab. Code 246 caps use at "40 hours or five days", exactly what handbook v2 grants, so under precedence rule 5 the statute controls and the handbook concurs. But whether the statute compels the specific use those scenarios turn on depends on 246.5, another section.
That chain was stopped rather than followed. conflict-019 now carries acceptable_authorities: [state, company]: the answer is determinate, the controlling layer is not, and the corpus cannot resolve it. This is the third time ingesting one section revealed a dependency on another, and following it indefinitely would make the corpus grow without the eval ever asking for it.
Ohio is 4 of 8 verified. The four that remain are marked "searched, not found, scope stated", which is a weaker standard and is labelled as such. Two open boxes concern New York provisions no scenario turns on; they stay open rather than being closed on assumption. One concerns California's pre-2024 sick leave minimum, which stopped mattering when the superseded slice moved to Ohio.
codes.ohio.gov has no programmatically reachable search, so a full-code keyword sweep is unavailable. Every finding above came from guessing the right chapter. The fourth contrary provision may sit in one nobody guessed.
Status: decided, Phase 5.5. Supersedes the DL-14 and DL-16 conclusions in DL-18.
Re-running the four configurations against the corrected corpus moved two decisions. The ground-truth errors were not cosmetic: they were changing what the experiment concluded.
| config | recall@10 | recall@3 | headroom |
|---|---|---|---|
voyage-law-2 / structure |
0.895 | 0.825 | 7.0 pts |
voyage-law-2 / fixed |
0.877 | 0.763 | 11.4 pts |
voyage-2 / structure |
0.877 | 0.702 | 17.5 pts |
voyage-2 / fixed |
0.842 | 0.675 | 16.7 pts |
DL-1 is unaffected and stands. The legal model wins in both chunking arms, by +1.8 and +3.5 points of recall@10 and by +12.3 and +8.8 of recall@3. Its advantage is if anything clearer than before.
DL-14 no longer gives one answer. Before the corrections both models put structure ahead by exactly 1.8 points, and the 2-point tie-break selected fixed in both. Now voyage-law shows +1.8, still inside the tie-break, while voyage-2 shows +3.5, outside it and selecting structure. The rule points two ways depending on a variable it was not written to consider.
DL-16 flipped for the configuration DL-14 selects. Headroom under voyage-law with fixed chunking was 9.6 points and is now 11.4, above the 10-point threshold. Under structure it is 7.0 and below.
Applying both rules mechanically to the adopted model gives: adopt fixed-size (gap 1.8, inside tie-break), then build a reranker (headroom 11.4, above threshold).
That outcome contradicts the reason the tie-break exists. DL-14 preferred fixed on the explicit grounds that structure-aware is the more complex implementation and complexity has to be paid for in measured benefit. Following the rules produces a system carrying a reranker, which is a second model, a second call per query, and additional latency and cost, in order to recover headroom that choosing the other chunker removes for free.
Fixed plus a reranker is strictly more complex than structure alone, and scores worse before the reranker is built (0.877 against 0.895).
This is not moving a threshold after seeing data, which is the failure pre-registration exists to prevent. The threshold stands: 1.8 points of recall@10 is one scenario and does not, on its own, justify a subdivision regex and a packing heuristic.
The tie-break assumed the two options were otherwise equivalent, and they are not. Choosing fixed triggers a second pre-registered rule that mandates strictly more complexity than the thing the tie-break was avoiding. Both rules were written to serve one principle: buy complexity only where it is measured to pay. Read together rather than in sequence, that principle selects structure-aware.
Recorded plainly so it can be judged: a rule I wrote in advance pointed at fixed, and I am not following it. The justification is the interaction, not the number, and if that reasoning is wrong the record shows exactly what was overridden and why.
voyage-law-2, unchanged.Before the corrections, the two models agreed at +1.8 and the decision looked settled. After, they disagree and the reranking rule flips. Five ground-truth errors were quietly steering the experiment, and the corrections were the difference between a clean-looking wrong answer and a messy correct one.
mrr fell across every configuration, from 0.723 to 0.674 at the best. That is expected and correct: conflict-006 now requires N.Y. Workers' Comp. Law 203, a section added to the corpus hours ago and one the system has to find rather than being handed the more prominent 204 it was previously credited for.
Status: decided, Phase 6.2. Retracts the "oracle-filter" framing in DL-17 and DL-18.
DL-16 committed query rewriting to triage and gave it three jobs: jurisdiction extraction, temporal resolution, and topic normalisation. Before implementing them I checked each against the 92 scenarios. Two do not survive.
The superseded slice settles it. superseded-001 and superseded-002 are word-identical questions with identical employee context, differing only in as_of_date: one 2023, one 2026. The same holds for -003/-004 and -005/-006. No amount of reading the question distinguishes them, because the distinguishing fact was never written in it.
The date is an input, the way today's date is an input to any HR system, and the spec already treats it as one: the demo exposes it as a picker. Exactly one question in the set contains a past-period expression, superseded-007, "I was told last year I only had three sick days. Has that changed?", and its correct as_of is unchanged, because it asks what the position is now.
So date extraction is not built. Building it would have added a component with no scenario able to hold it to account, whose only possible effect on this set is to move a correct date to a wrong one.
DL-16 justified it by saying the filter would otherwise "silently use today and answer a question about 2023 with 2026 law". That risk is real in general and absent here: nothing asks about 2023 without being told to.
75 scenarios supply the state in employee_context; 17 withhold it. Not one of the 17 names a state in its question. The extraction path therefore has zero coverage.
It is implemented anyway, because the demo has a free-text box and a reviewer will type "I work in California", and supplied context wins where the two disagree. But it is unmeasured, and this entry is the record of that rather than a coverage claim. Where jurisdiction is genuinely absent the correct behaviour is not to guess it: it is to decide whether the answer varies by state, which is a routing decision and is what the ambiguous and control slices measure.
DL-17 and DL-18 both carry a standing caveat that Phase 5's figures are oracle-filter numbers because jurisdiction and date "come from scenario metadata, not from the question", and that end-to-end will be lower by whatever the missing component costs. run_retrieval.py says the same in its docstring, and so did CLAUDE.md.
That is too strong. employee_context is defined in the schema as "what the asker has volunteered", and as_of_date is the date the question is asked. Both are caller inputs, not ground truth leaked into the query. An HRIS knows which state an employee works in and what today's date is. Passing them is not privileged information; it is the ordinary operating condition.
The check that settles it: of the 57 scoreable scenarios, 47 supply the state and 10 withhold it, and retrieval already runs unfiltered on those 10. There is no scenario where the harness applied a filter the running system could not.
I am recording this carefully because the correction runs in my favour, which is the direction that deserves more scrutiny rather than less. What genuinely remains between these numbers and end-to-end:
eval/baseline_retrieval.json exists precisely to catch it.answer. A mis-route to clarify, refuse or escalate means the scenario never retrieves at all, a failure the retrieval-only number cannot express.So the honest label is raw-query numbers with ordinary inputs, not oracle-filter numbers. The gap to end-to-end is the two items above, and both are measured rather than asserted.
Routing, and normalising the asker's vocabulary into the corpus's. That was always the substantive part: "grandma" has to reach "grandparent" and "time off" has to reach "family care and medical leave", and neither sparse matching nor a dense embedding does it reliably on a corpus of 300 chunks.
Structured output comes from a forced tool call rather than parsed prose. Asking for JSON in a prompt returns JSON in a markdown fence often enough that the fence-stripping becomes load-bearing, and then one day prose arrives before the fence and the strip returns something unparseable.
The query deliberately excludes state names and dates. Both are hard filters, and repeating them in the query text would turn a constraint into a ranking signal, which is the thing store.py is built to avoid.
Upgrade rule, unchanged and fixed in advance: below 0.80 macro-averaged route accuracy, this node moves from Haiku to Sonnet and the change is recorded. Above it, Haiku stays.
Cost: one full run over 92 scenarios is roughly 109,000 input and 8,000 output tokens, about $0.15 at Haiku 4.5 rates. Decisions are cached on disk by prompt and question, so re-runs are free and each prompt edit costs one run.
Status: decided, Phase 6.2. Haiku stays. Route accuracy 0.815 macro.
| run | change | macro | answer | clarify | refuse | escalate |
|---|---|---|---|---|---|---|
| v1 | first prompt | 0.725 | 0.667 | 0.400 | 1.000 | 0.833 |
| v2 | scope contradiction removed | 0.782 | 0.895 | 0.400 | 1.000 | 0.833 |
| v3 | clarify guidance made symmetric | 0.815 | 0.895 | 0.533 | 1.000 | 0.833 |
The upgrade rule fixed in advance was: below 0.80 macro, move this node to Sonnet. v1 came in at 0.725 and the rule fired. I did not upgrade, and the reasoning matters more than the number.
Sixteen of v1's twenty-nine failures were one shape, answer routed to refuse. Reading the model's own stated reasons showed it was executing my prompt correctly:
"Does the company pay me for jury service?" → "questions about compensation or pay are outside the scope of leave policy" "Do I keep my health insurance while I'm on family leave?" → "health insurance benefits and plan design are outside the scope"
The prompt listed "jury duty and voting" and "paid time off" as in the index, then said the index contains nothing about "pay" or "health plan design". Group health continuation during leave is 29 CFR 825.209, confirmed in DL-9 and sitting in the corpus. The prompt asserted P and not-P, and the model picked the exclusion.
So 0.725 measured a self-contradicting prompt, not Haiku's ability. Upgrading to Sonnet there would have bought a smarter model to resolve my contradiction and credited it with a fix that belonged to a bug.
The obvious hazard is that "the prompt was bad, let me try again" is exactly how someone iterates until they beat a threshold. The constraint adopted before touching anything, and the thing to judge this entry by:
A prompt may only be changed to remove a contradiction with the spec or with itself. No coaching drawn from which scenarios failed.
Both changes qualify and both are statable without reference to the score. The first is a flat self-contradiction. The second is a faithfulness failure: the section was titled "CLARIFYING IS A FAILURE WHEN IT IS UNNECESSARY" and warned only against over-asking, while the spec and DL-5 make both directions scored failures. Stating the rule symmetrically is restoring the spec, not tuning toward the answer. Both defects were found the way DL-15's largest finding was, by reading the artifact against the spec.
Iteration stopped at v3 on crossing the threshold, which is the moment the incentive to keep going is strongest.
How thin the pass is, stated plainly. 0.815 clears 0.80 by 1.5 points. One clarify scenario is worth 1/15 of that route and therefore 1.67 points of macro, so a single scenario flipping would put this below the threshold. It is a pass, not a comfortable one, and it should not be quoted as though the margin were meaningful.
Clarify remains the weakest route at 0.533, with under-clarification at 0.467: seven ambiguous scenarios still get answered when they should ask. Over-clarification stayed low throughout (0.039 → 0.052), so making the rule symmetric did not simply trade one failure for the other, which was the risk.
Running triage's rewritten queries through retrieval, against the baseline frozen before the agent existed:
| slice | baseline | rewritten, first run |
|---|---|---|
| adversarial | 1.000 | 0.500 |
| control | 1.000 | 0.900 |
| straightforward | 0.941 | 1.000 |
| superseded | 1.000 | 1.000 |
| conflict | 0.722 | 0.722 |
| overall | 0.895 | 0.877 |
The overall figure moved 1.8 points and reads as noise. Per slice, adversarial had lost half its scenarios. That is the exact failure mode the gate was written for, and it appeared on the first run rather than as a hypothetical.
Two of the four regressed scenarios had the literal string "none" as their search query, and they were exactly the two triage had mis-routed to refuse. Having decided not to answer, the model wrote a null query into a field the tool schema marks required, and retrieval searched for the word "none".
run_rewrite_check claims to isolate the rewrite from routing by scoring all 57 scenarios regardless of route. It cannot. Both come out of a single model call, so a bad route poisons the query it produced. The isolation is a property of the harness, not of the system.
Fixed in code rather than in the prompt: a degenerate query falls back to the raw question, which is never worse than a null one. Prompt untouched, per the constraint above.
| slice | baseline | rewritten, after fallback |
|---|---|---|
| adversarial | 1.000 | 1.000 |
| control | 1.000 | 1.000 |
| straightforward | 0.941 | 1.000 |
| superseded | 1.000 | 1.000 |
| conflict | 0.722 | 0.722 |
| overall | 0.8947 | 0.9123 |
Query rewriting improves retrieval by 1.8 points, and DL-16 predicted the mechanism correctly: straightforward-012 asks "can the company ask for a doctor's note", the rewrite supplied "medical certification", and 29 CFR 825.305 became retrievable.
One case it makes worse, and the slice total hides it. conflict-003 asks "I've been here 14 months. Can I take parental leave when my son is born?" and the rewrite dropped the figure, producing "parental leave eligibility for birth of child". Fourteen months is what makes 29 CFR 825.110 the target. The prompt already says to keep exact figures; the model did not. That is a real limitation of Haiku on this task and it is left in rather than prompted around, because the fix would be coaching drawn from a failure. It nets out against conflict-010, which the rewrite fixed, so the slice reads as unchanged while two scenarios moved in opposite directions.
Conflict still fails the gate at 0.722, correctly. The gate demands that slice improve, and resolve in Task 6.4 is what closes it. Triage was never going to.
Cost: three runs of 92 scenarios, roughly $0.73 total. Decisions cache by prompt and question, so the rewrite check re-ran for nothing.
Superseded in part by DL-26. The figures below are what this run measured, and they stand as the record of it. A later fix replaced two implementations of the naive resolver with one, which moved the baseline from 0.632 to 0.649 and the delta from +24.6 to +22.8. The conclusion did not change. The current headline number is +22.8.
Status: decided, Phase 6.3 and 6.4. Precedence correctness 0.877, naive baseline 0.632.
Same retrieval, same 57 answer scenarios, same run. The only difference is what decides the controlling authority: the layer of the top-ranked passage, or the precedence rules.
| slice | n | naive | precedence as code | delta |
|---|---|---|---|---|
| adversarial | 2 | 0.500 | 0.500 | +0.0 |
| conflict | 18 | 0.500 | 0.833 | +33.3 |
| control | 10 | 0.300 | 0.900 | +60.0 |
| straightforward | 17 | 0.765 | 0.882 | +11.8 |
| superseded | 10 | 1.000 | 1.000 | +0.0 |
| overall | 57 | 0.632 | 0.877 | +24.6 |
The handbook is the top-ranked passage 26 times out of 57. The spec's opening claim, that the correct answer frequently contradicts the most semantically relevant document, is no longer a premise. It is a measured 24.6 points, concentrated exactly where predicted: conflict and control.
The two slices with zero delta are the two where naive is already right. Superseded is solved by the effective-date filter, not by reasoning, and adversarial has two scenarios.
The model is asked two things: what does each layer's text say, and which provision is more generous to the employee. It is never asked which layer controls.
That decision is agent/precedence.py. Rules 1, 2, 4 and 5 are a pure function of the findings. Rule 3, effective dating, is not implemented there at all: the store already applies the as-of date as a hard filter inside each prefetch, and a second implementation would eventually disagree with the first.
The split is worth more than tidiness. Rule 5 is the subtlest rule in the spec and the one a review found missing entirely (DL-7), and in code it cannot be argued out of by a persuasively worded question, produces the same answer every time, and records which rule fired. concurrence_tie_break fired 4 times, statutory_floor 11, policy_may_exceed 8, silence_is_not_permission 33.
One integrity check earns its place: a layer that denies may never be ranked more generous than one that grants. It is the only way a model-supplied ranking can be checked against itself, and without it a handbook granting leave would lose to a statute that merely does not require it.
named the beaten source came back 0 of 8 while precedence read 0.895 and every individual resolution looked right.
The cause: the model was returning entire passages in the citation field. The prompt said to copy the citation "exactly as it appears in the passages", and passages are rendered as [citation] heading / body, so it copied the block. Nothing failed. Precedence ran, resolutions were produced, the trace looked plausible, and the citations were paragraphs.
This is the pattern this project keeps rediscovering. DL-12: every artifact agreed with every other and all were wrong. The only reason it surfaced is that a secondary metric was zero when it should not have been, and a flat zero is the one value too suspicious to ignore.
Fixed in two places, because either alone is insufficient:
resolve_citation validates against the citations actually retrieved, and rejects anything else. Longest match wins, because Cal. Gov. Code 12945 is a prefix of Cal. Gov. Code 12945.2 and both are in this corpus, which is DL-12's other trap. A citation that was never retrieved makes its layer silent rather than letting an entitlement rest on a provision nobody produced.The code half matters more. No amount of prompting reliably stops a model citing what it was not given, and a model cannot be trusted to police that in itself.
The correction moved the headline down, 0.895 to 0.860. The honest reading is not that the fix hurt: it is that 0.895 was never a valid measurement, because 0 of 8 proves the evidence it rested on was malformed. There is no regression to explain, only a first real number.
| prompt | precedence | note |
|---|---|---|
| resolve-v2 | 0.860 | citations validated |
| resolve-v3 | 0.860 | tie guidance made symmetric |
| resolve-v4 | 0.877 | tool schema aligned with the prompt |
The spread is one scenario. It should not be read as improvement. v3 did what it was written to do, cutting spurious ties from 4 to 1, and bought no accuracy; the errors moved rather than reduced. v4 removed a genuine self-contradiction, the tool schema still calling ties "common" while the system prompt said to tie only on identical provisions.
Reporting 0.877 as the result of three rounds of tuning would be dishonest. The measurement's precision at n=57 is about one scenario, so the finding is "roughly 0.87", and the three versions are indistinguishable. The constraint from DL-22 held throughout: prompts changed only to remove a contradiction with the spec or with themselves.
named the beaten source is 3 of 8. An answer that overrides the handbook without saying so leaves a reader who has already read the handbook unable to reconcile the two. resolve names the losing citations; whether the answer uses them is Task 6.5, and this number is the floor it starts from.
All 7 remaining failures are generosity comparisons, which is precisely the one judgment delegated to the model. Three hand a case to company that should be statutory and three do the reverse. No structural fix is available inside the current split: the rules are right and the evidence going into them is sometimes wrong.
Reachable after routing is 51 of 57. Six answer scenarios never get here because triage routes them elsewhere, and precedence on the reachable subset is 0.863. End-to-end is the product of both, and Task 6.7 reports it as such rather than quoting either alone.
Clarify needs no model. Triage has already named the missing fact, so turning tenure_months into a sentence is a five-way lookup. A model there would add cost, latency and a way for the question asked to drift from the fact recorded, which nothing downstream would catch.
Cost: four precedence runs and three triage runs, about $2.31 in total. The naive baseline is computed on the same pass from the same retrieval, so it costs nothing and cannot drift from the agent it is compared against.
Status: open, written 2026-08-26 before any open-model result exists. Resolves after Phase 7.
Written now, while the answer is unknown, for the same reason as DL-1 and DL-14. A comparison designed after seeing which side won is not a comparison.
Every enterprise buyer of an HCM system eventually asks whether it can run on cheaper or self-hosted infrastructure. This project has a 92-scenario harness and a frozen baseline, so the question is measurable rather than a matter of opinion.
Does an open-weights model reach the same precedence correctness and route accuracy as Haiku 4.5 on this task?
The two model calls in this system are deliberately narrow. triage classifies into four routes and rewrites a query; resolve reads passages and reports what each layer says. Neither is asked to reason about precedence, because that is code. Narrow extraction tasks are where smaller models are most competitive, so the split this project already made is the thing that gives an open model a real chance.
Against that: all seven remaining precedence failures are generosity comparisons, which is the single hardest judgment in the pipeline and the one most likely to degrade first.
Adopt the open model if, on the same scenario set and the same retrieval:
Two scenarios rather than one because DL-23 established that the measurement's precision here is about one scenario, and a threshold inside the noise would decide on nothing.
Otherwise keep Haiku and report the gap as the finding. An open model that costs nothing and loses five points is a legitimate answer to the buyer's question, not a failed experiment.
Retrieval, chunking, embeddings and the precedence rules are all held constant. The embedding model stays voyage-law-2 regardless: DL-1 measured that separately and it is a different decision.
Zero in dollars, on a free hosted tier. Not zero in time: every cached decision is keyed by model, so both arms re-run in full.
Phase 6 finishes on Haiku first, and this runs afterwards. Switching mid-build would invalidate DL-22 and DL-23, including the naive-versus-agent comparison that is the project's headline result, and re-establishing them is more expensive than the experiment is worth at this point.
Threshold 1 met. openai/gpt-oss-120b scores 0.818 macro route accuracy against Haiku's 0.815, across all 92 scenarios, at $0.00 versus $0.26.
| route | n | Haiku 4.5 | gpt-oss-120b |
|---|---|---|---|
| answer | 57 | 0.895 | 0.947 |
| clarify | 15 | 0.533 | 0.800 |
| refuse | 14 | 1.000 | 0.857 |
| escalate | 6 | 0.833 | 0.667 |
| macro | 0.815 | 0.818 | |
| under-clarification | 0.467 | 0.200 |
+0.002 is a tie, not a win. It sits far inside the one-scenario noise floor DL-23 established, and calling it an improvement would be the overclaiming this log exists to prevent. What is real is the shape: the free model is much better at clarify, the route Haiku was worst at and the one DL-5 exists to test, and worse at refuse and escalate. Read those cautiously too: at n=6, one escalate scenario is worth 0.167.
Threshold 2 could not be measured, for a structural reason rather than a quality one.
resolve prompt tokens |
median 5,385, max 9,036 |
| Groq free-tier per-request ceiling | 8,000 |
| prompts over it, all 57 | 13 (23%) |
| prompts over it, conflict slice | 7 of 18 (39%) |
The open model never gets to attempt the reasoning step. The request is larger than the tier accepts. This is not "it reasons worse", it is "it is never offered the evidence".
Scoring the 11 conflict scenarios that do fit was considered and rejected. All seven excluded carry the full ten passages, and prompt size is a proxy for how much law bears on the question, which is a proxy for how hard the precedence call is. The subset is self-selected toward easier cases and would flatter the open model. A biased number is worse than an honest gap.
Five constraints, none documented where they could be planned for, each found by hitting it:
max_tokens and emitted before the tool call. A budget sized for the answer truncated the call one closing brace short, rejected server-side.max_tokens is charged against the rate limit, not the tokens produced. Reserving 1,536 where 278 were used throttled throughput fivefold for nothing.openai/gpt-oss-20b cannot honour tool_choice, returning "Tool choice is required, but model did not call a tool". Disqualified on the contract rather than on its score, which was the acceptance criterion fixed before any of this ran.A sixth is ours rather than theirs: concurrent foreground calls share the same bucket. Probing while a run was in flight caused six multi-minute 429 stalls consuming 57 of 74 elapsed minutes.
Routing is free. A 120B-class open model matches a paid frontier model on classification and query rewriting, and beats it on the hardest route, at zero cost. To a customer asking whether triage can run on their own infrastructure, the answer is measured, and it is yes.
The evidence step is not, on a free tier. Not because of the model: 23% of prompts exceed the per-request ceiling and the daily cap is half what one arm needs. Answering it properly needs a paid tier or self-hosting with a larger context budget. Until then it stays unmeasured rather than half-measured.
retrieve, so the arms would not have sat on byte-identical retrieval even had the run completed. DL-28 measured that effect at zero, which helps and is not nothing.Status: decided, Phases 6.5 to 6.7. Fully correct 0.587, up from 0.293 once the measurement was fixed.
All 92 scenarios through the whole graph, nothing held at its correct value.
| metric | value |
|---|---|
| route accuracy, macro | 0.815 |
| precedence correct (n=57) | 0.772 |
| passed verification | 0.717 |
| forbidden citation leaked | 0.011 |
| required citations present | 0.491 |
| named the beaten source (n=8) | 0.125 |
| fully correct | 0.587 |
| slice | n | route | fully correct |
|---|---|---|---|
| out_of_scope | 12 | 1.000 | 1.000 |
| superseded | 10 | 1.000 | 0.900 |
| straightforward | 17 | 1.000 | 0.706 |
| adversarial | 10 | 0.800 | 0.700 |
| ambiguous | 15 | 0.533 | 0.533 |
| control | 10 | 0.500 | 0.300 |
| conflict | 18 | 1.000 | 0.167 |
Fully correct means all of it at once: right route, right controlling authority, required citations present, nothing forbidden, and grounded. It is the only number a user would recognise, and it is far below every component score, which is the honest shape of an end-to-end measurement.
Precedence drops from 0.877 isolated to 0.772 end to end. The difference is the six answer scenarios triage routes elsewhere, which never reach the resolver at all.
verify never runs on a refusal, and I scored that as failing verification. 34 of 92 scenarios correctly route to clarify, refuse or escalate. None reach verify, because none assert an entitlement and there is nothing to ground. verification is None was read as "did not pass", so every correct refusal counted against the system.
The tell was a slice reporting route 1.000 and fully-correct 0.000 in the same row. out_of_scope got every routing decision right and scored zero. That combination is not a result, it is a contradiction, and it went from 0.000 to 1.000 the moment "did not run" stopped meaning "failed". Fully correct went 0.293 to 0.587 on this one fix.
The figures check failed valid answers two ways. It flags numbers in the answer that do not appear in the sources, and both false positives were found by it rejecting correct answers rather than by reasoning about it.
[29 CFR 825.201] that then refers to "section 825.201" leaves a bare number behind, which was reported as an unsupported figure. Numbers appearing inside any known citation are now excluded.Both bugs share a shape with DL-12 and DL-23: nothing raised, every artifact agreed, and the only signal was a number too extreme to be true.
29 scenarios are missing a required citation, and 22 of those are a cascade. Verification failed, the answer was replaced with a referral, and the citation went with it. Of the 7 genuine misses, 6 also had the wrong controlling authority, so the citation is absent because the agent chose the wrong layer rather than because composition forgot to cite.
So required citations present is not measuring compose. It is measuring how strict verify is, and reporting it as a composition score would be wrong.
Verification strictness is dominated by self-grading. 20 of 26 verification failures are the entailment check, which currently runs on the same model that wrote the answer. That is precisely the self-grading the spec warns against, kept deliberately (see the verify module docstring) because DL-24's open-weights arm supplies a cross-family verifier for free. Whether a different model is less trigger-happy is now a measurable question rather than an assumption.
Conflict is 1.000 on routing and 0.167 fully correct, the widest gap in the set. Every conflict scenario reaches the resolver and most get the authority right; they fail at grounding. This is the slice the whole project is about, so it is also where the next work goes.
named the beaten source is 1 of 8. resolve names the losing citation and compose is instructed to address it, and it still mostly does not survive to the answer. An answer that silently overrides the handbook is unusable to the person holding the handbook, and this remains the weakest requirement in the system.
One forbidden citation leaked, conflict-012 citing LEAVE-005. A rate of 0.011 across 92. The store filters superseded and wrong-jurisdiction text before retrieval, so this is a provision that was legitimately retrieved and should not have been relied on.
clarify, refuse and escalate are all deterministic. Triage has already decided the route and written the sentence explaining it; passing that through a second model would add cost and let the explanation drift from the decision it explains. Four of verify's five checks are functions for a stronger reason: code cannot share a blind spot with the model that wrote the answer.
agent/build.py assembles the graph in one place, and build_baseline is the same graph with precedence swapped for "trust the top-ranked passage". Phase 9's comparison is therefore one code path with one component changed, not two implementations that could differ for reasons nobody intended.
Cost: the full end-to-end run is about $0.57 cold and $0.13 warm. Total project spend to here is roughly $3.00.
Status: decided, Phase 6.8 hardening. Independent review of Phase 6, 14 findings, all addressed.
Every eval scorer could be mutated to return True unconditionally and the full suite stayed green. precedence_correct, naive_correct, addressed_what_it_should, verified, required_present, forbidden_present, and build_baseline swapped to use the real resolver: seven mutations, seven survivors, 303 of 303 tests passing throughout.
The agent code was mutation-tested from Task 6.1 onward. The code that measures it was not, so both halves of the headline comparison and all of fully_correct rested on arithmetic nothing tested. DL-25's verified fix, the bug that halved the headline, had no regression test either.
This is DL-10 one level up. The lesson there was "assert values, not relationships"; the lesson here is that the measuring apparatus is code and needs the same discipline as the thing measured. A wrong number and a right one are indistinguishable from inside.
tests/test_scorers.py now exists. All twelve review mutations are caught.
| metric | before | after | why |
|---|---|---|---|
| passed verification | 0.717 | 0.621 | denominator was 92; only 58 reach verify |
| fully correct | 0.587 | 0.620 | net of five scorer and checker fixes |
| conflict, fully correct | 0.167 | 0.278 | subsection citations no longer failed |
| naive baseline, precedence | 0.632 | 0.649 | one implementation instead of two |
| precedence delta | +24.6 | +22.8 | same |
passed verification at 0.717 was wrong and I reported it. 34 scenarios correctly route to refuse, clarify or escalate and never reach verify; the metric counted them as passes. This is the identical "did not run is not a pass" conflation DL-25 fixed for fully_correct and left standing one line away, which is worth recording as a pattern rather than a slip: fixing an instance of a bug is not fixing the bug.
The figures check compares quantities in the answer against the sources. It had one hole in each direction and one that made it a rubber stamp.
29 was exempt corpus-wide. The check excluded any digit appearing inside a retrieved citation, and every federal citation is 29 CFR .... An answer claiming "you are entitled to 29 workweeks" passed. Confirmed by the reviewer, not hypothetical.
Number words were matched as substrings, so "written" contains "ten" and marked 10 as supported, and "none" contains "one". Any corpus of ordinary prose therefore supported those figures unconditionally.
And it failed correct answers: 1,250 against 1250, and bare section references like "section 825.201".
All three came from the same mistake: extracting every number and then trying to subtract the ones that were not quantities. The check now requires a unit. A section number is never followed by "weeks"; a fabricated entitlement always is.
Check 1 could never fire. It compared state["citations"] against the retrieved set, but compose had already filtered that list by the identical predicate. Its comment said it caught citations "appearing in the prose", which is behaviour the code did not have. It now reads the prose.
build_baseline hardcoded its verify model while build_agent took a parameter. Identical today, but DL-24 pre-registers swapping in a cross-family verifier, at which point the agent would get it and the baseline silently would not, on exactly the Phase 9 screen that is the project's argument.
There were two implementations of the naive baseline, one in build.py and one in the precedence scorer. DL-23 claimed the baseline "cannot drift from the agent it is compared against"; it could drift from itself. One implementation now, and the baseline moved by a scenario when they were unified, which is the drift made visible.
statutory_floor labelled both rule 1 (federal against state) and the handbook-below-the-floor case. Those are different rules and one of them is the project's thesis. Split into policy_below_floor, and the counter now reads 8 handbook-below-floor against 3 federal-versus-state, where it previously reported 11 undifferentiated.
Fixing DL-12's prefix trap in the scorers (Cal. Gov. Code 12945 is a prefix of 12945.2), I required citations to be bracketed. That broke the other direction: compose brackets the controlling provision and names the beaten handbook in prose, so named the beaten source reported a flat 0.000.
DL-23 says a flat zero is the one value too suspicious to accept, and it was right twice in one project. Boundary matching handles both without dictating how an answer is written.
And the check was failing subsections. The model writes Cal. Gov. Code 12945.2(b)(13) where Cal. Gov. Code 12945.2 was retrieved, which is a more precise pointer into the same passage. That accounted for 14 of 30 verification failures and was wiping correct answers. A cited string now resolves if it extends a retrieved one with a bracket, so the allowance cannot launder 12945.2 in on the strength of 12945.
named the beaten source is 0 of 8, and it is real this time. Six of the eight fail verification and degrade to a referral; the two that survive do not mention the source they override. An answer that silently contradicts the handbook is unusable to the person holding it, and this is now the weakest requirement in the system by a distance.
Under-clarification is 0.467, unchanged, and caps ambiguous at 0.533 in both the agent and the baseline.
The precedence rules match the spec, including rule 5's ordering above rule 2 and its deliberate refusal to order federal against state; the reviewer traced eight configurations by hand. The denies versus grants integrity check is exactly sufficient. DEFECTS.md remains unreachable through both guards and through a rename attack. 26 of 30 mutants in agent/ were already killed. Every figure quoted in DL-21 through DL-25 reproduces exactly against the saved runs.
Final Phase 6 numbers: fully correct 0.620, precedence 0.772 end to end and 0.877 isolated, verification 0.621 over the 58 that reach it, route accuracy 0.815 macro, one forbidden-citation leak in 92.
composeStatus: decided, Phase 6.9. named the beaten source 0.000 to 0.250. Side effects across three other metrics.
The requirement: when the answer overrides the handbook, it has to say so. A reader holding the handbook cannot reconcile an answer that silently contradicts it, however correct that answer is. Eight scenarios carry must_address and the system scored zero.
The obvious reading was that compose ignored its instruction. That was wrong, and it was wrong three separate ways.
resolve was naming the wrong sourceprecedence.py builds non_controlling_to_address from "layers that spoke and lost", which is usually a federal section. must_address in the ground truth is always a handbook policy. So the system was diligently telling readers about CFR provisions they never opened, while omitting the handbook they had.
Worse, one case was unreachable by construction. conflict-005 asks about a sick grandparent: federal denies, Ohio adds nothing, and the handbook is silent. A silent layer never enters the comparison, so it can never be a loser, and yet "the handbook does not cover this" is exactly what the reader is trying to establish.
Fixed by reading the retrieved passages directly rather than the findings. Handbook policies come first, ahead of any statute that merely lost.
conflict-004, -005 and -010 failed for the plain reason that no company chunk was in the top ten. A source that was never retrieved cannot be addressed, and no amount of prompting fixes that.
retrieve now searches deeper than it returns and tops up with a handbook passage when none ranked. Appended, never substituted, so nothing statutory is displaced and the ranking the regression gate watches is untouched. The gate confirms it: recall@10 held at 0.9123, no slice regressed.
The principle is worth stating on its own: the handbook is always relevant to an employee leave question, whether or not it controls, because it is the document they have already read.
compose to doAny source the model omits now gets a deterministic note built from the citation and the rule that defeated it, both of which the resolution already holds. No new claim is asserted. Same reasoning as appending the disclaimer rather than trusting the model to write one: a requirement that can be forgotten will be.
The trace records addressed_by_model and addressed_by_fallback separately, so enforcing the requirement does not destroy the measurement of whether the model met it unaided. It did, 8 times out of 9.
| of 8 | |
|---|---|
| named in the final answer | 2 |
| named in the draft, then wiped when verification replaced the answer | 2 |
| a different handbook policy surfaced, or none | 4 |
So the ceiling is no longer compose. It is two other things:
Verification strictness. A third of answers are replaced by a referral, taking their citations with them. Nothing downstream of compose can be improved while that holds, and 20-odd of those failures come from entailment self-grading on the model that wrote the answer. DL-24's open-weights arm is the pre-registered test for exactly this, which is now the highest-value thing left in the project.
Which handbook policy is the right one. The top-up fetches a handbook policy, not necessarily the one the reader would have opened. That is a retrieval relevance problem, not a composition one, and it is not worth solving before the verification bottleneck is understood.
| metric | before | after |
|---|---|---|
| named the beaten source | 0.000 | 0.250 |
| precedence correct, end to end | 0.772 | 0.789 |
| required citations present | 0.491 | 0.544 |
| passed verification | 0.621 | 0.672 |
| fully correct | 0.620 | 0.620 |
| recall@10, regression gate | 0.9123 | 0.9123 |
fully_correct did not move, which is the expected shape: it is a five-way conjunction and the scenarios that gained on one clause were already failing another.
Fixing this created a circular import, compose needing the citation matcher from verify while verify reads the disclaimer from compose. That is the usual sign a helper belongs in neither, so agent/citations.py now owns citation matching. Both rules it encodes, the prefix trap and the subsection allowance, have been got wrong once each in this project, in opposite directions, and are now written down in one place.
Status: prediction written 2026-08-26 before the effect is measured. Result appended below.
The proposal: stage 1 pre-filter on indexed, selective fields; stage 2 ANN vector search; stage 3 post-filter on non-indexed attributes. It is a standard and correct production RAG pattern, and it was worth checking against this system rather than adopting or dismissing on reputation.
store.py carries payload indexes on jurisdiction, authority_layer, content_status, citation and both effective-date ordinals, and applies jurisdiction plus the as-of window as a hard filter inside each prefetch rather than after fusion. That placement is the subtle half: a filter applied after RRF is a post-hoc trim that can silently return fewer than k results with no error, which looks like a worse answer rather than a bug.
| Qdrant search, with pre-filter | 5.8 ms median, 10.8 ms p90 |
| Qdrant search, no filter at all | 5.6 ms median |
| Full graph, four live model calls | 13,198 ms |
| Retrieval share of end-to-end | 0.04% |
Filtering costs nothing measurable, and retrieval is four ten-thousandths of the time budget. At 307 chunks against an indexing threshold of 10,000 there is no ANN traversal to optimise. Perfecting retrieval latency would save 6 ms out of 13 seconds.
If latency is the goal the levers are elsewhere: four sequential model calls, of which verify runs only when the deterministic checks pass, and the pre-computed curated responses already planned for Phase 8, which take the reviewer's path to zero calls.
Recorded because the intuition was reasonable and the number settles it. A latency optimisation aimed at 0.04% of the budget is not an optimisation.
Not speed. Guaranteed presence. Some documents must be in the candidate set for the agent to reason correctly, whatever their similarity rank, and that is a non-indexed condition about the set rather than a filter on individual chunks.
One instance already exists and paid for itself: DL-27's handbook top-up, which is precisely an over-fetch-then-trim post-filter and moved three metrics.
Measurement identified a second, larger one. Five of 33 Ohio answer scenarios have an absence record reachable at rank 30 but not at rank 10. Rule 4, silence is not permission, depends on the agent distinguishing a recorded silence from a retrieval miss, which is the entire reason absence records are documents rather than config. It cannot make that distinction about a record it never saw.
A third candidate was measured and rejected: per-document diversity capping. One document takes four or more of ten slots in 7 of 57 scenarios, but only 2 of 57 have a required citation absent from the top ten and present in the top thirty. Two scenarios is the noise floor DL-23 established, and DL-14's precedent is that complexity has to be paid for in measured benefit.
Adopt the absence-record guarantee if precedence correctness improves by at least 2 scenarios and no slice regresses on the frozen retrieval baseline. Two rather than one because the measurement's precision here is about one scenario.
Prediction: it improves Ohio conflict and control scenarios, where the correct answer is "no state provision covers this, so the handbook governs", and does nothing elsewhere. What would falsify it: if surfacing more absence records makes the model read silence as denial, precedence correctness falls on scenarios where a layer genuinely denies.
Why post-filter rather than raise limit to 30: every extra chunk is roughly 300 tokens in the resolve and compose prompts, which are the actual cost and latency, and 20 extra passages would triple both to surface one document. Over-fetching cheaply and trimming deliberately is the whole point of the stage.
Revisit at scale. Below the indexing threshold Qdrant searches exhaustively, so pre-filter and post-filter selectivity are indistinguishable here. Past roughly 10,000 chunks HNSW engages, filter cardinality starts to decide whether a filtered traversal beats a full scan, and this decision should be measured again rather than assumed to hold.
Precedence correctness improved by zero scenarios. 45 of 57 end to end and 50 of 57 isolated, both unchanged. Every other metric flat to three decimal places: fully correct 0.620, verification 0.672, required citations 0.544, named the beaten source 0.250. No regression anywhere, and the frozen retrieval baseline held at 0.9123.
The pre-registered bar was two scenarios. It delivered none, so the absence-record guarantee is not adopted, and the code carries it as UNADOPTED_RECORDED_SILENCE, outside GUARANTEES, with a test pinning it out so re-adding it is a deliberate act with a number attached rather than a drift.
What the null result actually says. Five Ohio scenarios had a rescuable absence record, and surfacing it changed nothing. So the agent was not losing those scenarios for want of the record: it either already had enough to answer, or it was failing for an unrelated reason. The mechanism was available and the bottleneck was somewhere else, which is the most common way a plausible optimisation turns out not to matter.
The falsifier did not fire either. Surfacing more silence did not make the model read silence as denial, so the risk that argued against it was also not real.
The three-stage structure itself, and only the handbook guarantee inside it, which DL-27 had already adopted on measured benefit. The refactor is behaviour-preserving: same lookahead, same top-up count, same appended-never-substituted rule. What it buys is that stage 3 is now a named list of Guarantee objects with a stated reason each, so the next one is a data change with a threshold rather than another special case wired into the node.
Two of three candidate post-filters were rejected on measurement, one before building (per-document diversity, 2 of 57 rescuable, at the noise floor) and one after (absence records, zero effect). One was adopted and paid. That ratio is the normal one, and a pipeline where every proposed optimisation turns out to help is a pipeline whose measurements are not load-bearing.
The latency finding stands on its own and is the more useful half: retrieval is 0.04% of end-to-end time, so the staged pattern is worth having for correctness and worth nothing here for speed.
Status: decided, Phase 8.
The spec names six curated scenarios and calls the handbook-conflict case "the entire argument". Running it end to end showed a referral:
"I could not confirm an answer to this from the policies on record. Please ask your HR team."
conflict-001 is the canonical D-1 case: the handbook demands 18 months, CFRA grants at 12, she has 14. The precedence rules resolve it correctly. verify then rejects the composed answer and the whole thing degrades. It traces to the same self-grading entailment problem that produced 20 of 26 verification failures in DL-25.
The conflict button now shows conflict-002, and the reason is not that it passes.
| naive baseline | truth | |
|---|---|---|
| conflict-002 | state | company |
| conflict-004 | state | state |
| conflict-010 | state | state |
| conflict-021 | state | state |
Of the five conflict scenarios that are fully correct end to end, only conflict-002 and conflict-009 are ones the naive baseline gets wrong. The other three it gets right by luck, so the baseline toggle would show no difference and the demo's central comparison would be empty.
conflict-002 is also the better teaching case. The handbook grants ten paid bereavement days against a five-day statutory floor, so policy controls and the naive intuition fails in the harder direction: this system is not "always prefer the law", and D-2 exists precisely to punish an agent that learned that.
Choosing which of eighteen conflict cases to show is unavoidable; six buttons cannot show eighteen. What makes it honest is that the slice's measured score travels with the payload:
"slice_performance": {"n": 18, "route_accuracy": 1.0, "fully_correct": 0.278}
Read from eval/runs/end_to_end.json rather than restated, so the demo cannot quote a number the evaluation no longer produces. A reviewer sees one working example and, beside it, that the slice is 0.278 fully correct. Every payload also carries expected and matched_expectation, so a scenario the agent gets wrong displays as wrong rather than being quietly dropped.
The dishonest version of this is picking winners and implying the slice is solved. The honest version is picking a representative that demonstrates the mechanism while reporting what the slice actually scores.
The demo's headline case cannot be shown because verification rejects a correct answer. That is now the highest-value open item in the project, above anything in Phase 9, and it has a known cause and a pre-registered fix: DL-24's open-weights arm supplies a cross-family verifier, and verify prompts are small enough to fit the free tier's per-request ceiling even though resolve prompts are not.
Status: applied, Phase 8 review. Three confirmed criticals, 15 mutations now caught.
/api/ask invoked the graph and then recorded budget, with no try/finally. A run that raised after calling the model therefore spent real tokens and consumed nothing. Thirty such requests spent 3,000 input tokens with the global counter still reading 8 of 8.
So the sentence this phase was built around, that the global breaker is the last line, was false: it was only reached by requests that succeeded. Any reliably-triggerable mid-graph failure, and there are two obvious ones in the Groq token ceiling and a Qdrant timeout, was a source of unmetered inference.
The test named test_budget_is_charged_only_after_the_work_runs asserted remaining == before - 1, which is true whether recording happens before or after the work. It tested the arithmetic and not the ordering, and a mutation moving record above the call survived it.
DL-29 argued the demo stays honest because each payload carries its slice's measured score. That score was read from eval/runs/end_to_end.json, and eval/runs/ is gitignored. In any deployment every payload reported slice_performance: {}.
It failed silently: a missing file returned {}, and all 451 tests passed. The honesty argument was true only on the machine that generated it, which is the worst form of this failure because nothing looks wrong.
The scores are now a snapshot committed beside the records, written by the generator in the same pass that captures the runs, so the numbers cannot describe a different run from the answers they sit next to. A missing eval artifact makes the generator refuse to run rather than shipping records that quote nothing.
client_ip took the first X-Forwarded-For entry, on the reasoning that later hops are attacker-controlled. It is the other way round: each proxy appends, so the leftmost value is whatever the client sent. Rotating the header walked straight past both the per-IP and per-session limits; only the global cap stopped it.
Now Fly-Client-IP first, which the platform sets and a client cannot forge, then the rightmost forwarded hop, then the socket peer.
Unbounded key growth. Session ids and forwarded IPs are client-supplied and unauthenticated, so a caller can mint unlimited rate-limit keys; 500 refused requests allocated 1,000 permanent deques. Eviction now drops expired keys only, and never the global one: bounding memory must never hand out budget.
Cost was wrong. usd apportioned tokens across models by call share, so an identical request reported a different figure once a second model entered the process. Cost is now computed per model from that model's own tokens. Request accounting moved from subtracting snapshots of a shared counter to a ContextVar, because under concurrency the subtraction is simply wrong: eight simultaneous requests each spending 1,000 tokens reported 1,000 through 8,000.
Nine of the fifteen mutations survived the first round of fixes, because the tests I wrote alongside them asserted the wrong things: matched_expectation checked for the key's presence rather than its value, key growth asserted <= 500 which no-eviction also satisfies, staleness asserted only is False, and the scoped-usage test wrote to the ContextVar by hand instead of driving call.
Writing a test at the same time as the fix is not the same as writing a test that would fail without the fix. Mutation is the only thing that has reliably told the two apart in this project.
Status: decided, Phase 9.
The spec calls the baseline toggle "the single highest-leverage feature" and says the delta "is the entire argument". Building it ran straight into the constraint the rest of this project has been living with: both model providers were exhausted. Anthropic is out of credits and the Groq free tier's daily cap was spent.
Generating a baseline arm the obvious way, by running build_baseline end to end, needs compose and verify per scenario. It failed on the first call.
The claim is that a system trusting the most relevant passage picks the wrong authority, and one applying precedence picks the right one. That delta lives entirely in resolve, and the baseline's resolver is naive_resolve: take the top-ranked passage. It calls no model.
So the baseline arm records its resolution only: the authority it selects, the passage it selected it from, and whether that is correct. No composed answer.
Composing one would have added two model calls per scenario to restate a conclusion the resolution already shows, and it is precisely what could not be generated. The screen the spec calls the entire argument is now also the screen that cannot be blocked by an unfunded key.
| scenario | agent | rule | naive | naive correct |
|---|---|---|---|---|
| conflict | company | policy_may_exceed | state | no |
| straightforward | federal | none | federal | yes |
| superseded | company | none | company | yes |
| refuse / escalate / ambiguous | (declines) | none | (never resolves) | see DL-32 |
The conflict row is the one the spec asks for, and it fails in the harder direction: the handbook grants ten paid bereavement days against a five-day statutory floor, so policy controls and the naive system reaches for the statute.
The two rows below it were wrong and are corrected in DL-32. They described a code path the baseline never executes.
Every scenario shows the ground-truth expectation beside the result, the slice's measured score (conflict is 0.278 fully correct), and the overall figure in the footer: 0.620 fully correct across 92 scenarios, read from the committed snapshot rather than restated in the markup where it could drift.
Two of six scenarios show the baseline being right by luck, and they are labelled that way rather than dropped. A demo that only showed the four wins would be a brochure.
Self-contained: no CDN, no build step, no framework. Every dependency is another thing that can be unavailable when someone opens it, and a test asserts no external src or href survives in the served HTML.
Nothing reaches the DOM through innerHTML. The first draft used an escape helper covering &<>, which a security check flagged: it does not cover quotes and is one refactor away from being wrong on a public endpoint. Values now go in via textContent or a created node, and a test pins the absence of innerHTML.
Status: corrected, Phase 9 review. Retracts two rows of DL-31's table.
DL-31 reported that on refuse and escalate the naive baseline "selects federal and would answer confidently, where the agent declines", and called that finding better than the one the spec asked for.
It was false, and the generator produced it.
build_baseline is the same graph with only the resolver swapped. It therefore shares the agent's triage, and agent/graph.py gates retrieval behind the route: nothing downstream of triage runs unless the route is answer. On clarify, refuse and escalate the baseline declines identically and never reaches a resolver at all.
The generator called triage → retrieve → naive_resolve unconditionally, so it recorded an authority for three scenarios the baseline never resolves. The demo then painted those panels red and told the reader that a naive system would have answered a question it in fact refuses. Asking the live API the same question with baseline: true would have contradicted the page.
The comment directly above the offending lines said "the only variable is how the authority is chosen", which was true of the design and false of the code four lines later.
One scenario of six has a genuine baseline delta, not four. conflict, where the handbook exceeds the statutory floor so policy controls and the naive system reaches for the statute. Two are right by luck and were already labelled so. Three have no delta at all, and the page now says exactly that: both arms decline, precedence never comes into it.
Recorded as a value in a test, so a future change that produces more deltas fails and the number is updated deliberately rather than drifting upward.
save_baseline had no test. Mutating correct to True, and dropping each of the two expectation fields, all survived: the tests read committed JSON and never exercised the function that writes it. The logic was right; nothing proved it.
matched_expectation compared the route alone while the page labelled it "matches ground truth" in green. A run reaching the right conclusion from the wrong authority, which the spec calls luck rather than correctness, would have rendered as a match. It now checks route, authority and required citations.
The page's rendering had no tests. Six mutations survived, including forcing the baseline tag to "right, by luck" and blanking the footer's measured score. Every honesty element the page draws was unasserted; only the payload was tested. The checks added are structural, on the source, not render tests. A real render test needs a JS runtime this project does not have. They catch deletion and inversion, which is what the mutations did. They would not catch a CSS change that hides an element. Stated rather than glossed.
The helper written to cover save_baseline wrote over the committed conflict record and restored it in a finally that was initially a no-op. Once fixed, it captured the already-polluted file as its "original", so the corruption became self-sustaining. The tell was the test's own fake provenance sitting in shipped demo data.
The helper now redirects STORE to a temporary directory: removing the possibility rather than managing it. A test that writes to committed data poisons the repository even after the test is fixed.
Status: decided and implemented, Step 4. Effect projected from the last scored run; not yet re-measured (credits).
verify runs five checks and they are not the same kind of thing.
| check | kind |
|---|---|
| citations were retrieved | decidable |
| controlling provision cited | decidable |
| figures appear in sources | decidable |
| the answer cites something | decidable |
| claims follow from their sources | judgment |
The first four are settled by set membership and string search. A citation that was not retrieved was not retrieved, and no reading makes shipping it acceptable. The fifth is one model's reading of whether a sentence follows from a passage, and it was allowed to silently discard answers that passed all four.
Of 19 scenarios that reached verify and failed it, 15 had the correct controlling authority, and 9 of those were the conflict slice. The precedence machinery worked, the answer was drafted from the right provision, and a judgment call threw it away. That is why conflict scores 1.000 on routing and 0.278 fully correct.
This does not prove the verifier is biased, and the alternative reading is live: one flagged claim was "Federal law [29 CFR 825.200] provides twelve weeks of FMLA leave for birth or bonding", where 825.200 gives the twelve weeks and the birth-or-bonding link is 825.120. The verifier may simply be right that compose makes compound claims while citing one provision. Step 1 will settle it by hand. What the measurement does establish is that the verifier, not composition, is the binding constraint.
Decidable failures block. Judgment failures annotate: the answer ships with the claim flagged, in a structured advisories field rather than appended prose, because mixing a system caveat into the answer text makes the model's words and ours indistinguishable.
The reader argument. A referral saying "I could not confirm an answer to this" gives the reader nothing: no answer, no citations, no way to check. An answer with a flagged claim gives them the provision, the reasoning, and a specific sentence to be careful about.
VERIFY_ENTAILMENT_BLOCKS=1 restores the strict posture. The risk appetite is a product decision rather than an engineering one, and a real HCM deployment might well weigh an unsupported claim as worse than no answer.
Relaxing what blocks an answer raises fully_correct without improving anything. This project has caught itself at that move repeatedly, so it is not made silently.
Two numbers are now reported. fully_correct counts what blocks. fully_correct_strict counts advisories as failures and is the headline. By construction this change cannot move the headline: an answer that ships with a flagged claim is still a failure under the strict reading. What it changes is what the reader gets, which was the point.
The gap between them is itself the finding, and it is printed on every run: how many answers are correct in every decidable respect and carry a claim the verifier could not tie to its source. Projected from the last scored run, that is 15 of 92.
Re-running the evaluation needs live compose calls: the beaten-source fix in DL-27 changed one resolution, which changed its prompt, which invalidated its cache entry. Both providers are exhausted. The 15 figure is therefore a projection from the saved run, not a measurement of the new behaviour, and it is labelled that way until it can be re-run.
Status: Step 1 complete, by hand, against the ingested text. Overturns the plan in DL-33.
I had assumed verify rejects correct answers because it self-grades on the model that wrote them. Reading all fifteen rejected cases against their cited passages does not support that, and two things I had merged turn out to be different problems.
| failed a decidable check | 7 |
| failed entailment | 8 |
Six of the seven had no valid citation at all after compose ran. Those block correctly and DL-33's change does nothing for them. They are a composition failure, not a verification one, and I had been counting them toward a verifier problem.
Of the 14 flagged claims, roughly eight are ones the verifier was right to flag:
conflict-001: "...for birth of a child ... regardless of tenure". 29 CFR 825.120 says "eligible employees", and eligibility is 12 months plus 1,250 hours under 825.110. The claim is false.conflict-017: "federal law requires no service period for bonding leave eligibility" and "no hours or service threshold at all". Both false, and this is the scenario whose entire point is the twelve-month boundary.conflict-019, conflict-020: the retrieved chunks of Cal. Lab. Code 246 are the accrual amounts, not the qualifying uses. The answers assert what the cited text does not contain.A cross-family verifier would not have improved these. It might have lost them. Step 3's value drops sharply.
One plain error. superseded-003 claims "Ohio has no state paid sick leave statute" and cites the absence record, whose text reads "Ohio has no state paid sick leave statute requiring private employers to provide or accrue paid sick leave." The evidence was in front of it and it flagged the claim anyway.
Five where the evidence was withheld by design. verify builds cited_text from passages whose citation appears in the answer. So a claim resting on anything else has no support to be checked against:
conflict-001) and "federal law controls, not the company policy" (straightforward-011) are precedence conclusions. No single passage states them, because they are derived. The system's central output is, by construction, unverifiable against its own evidence set.conflict-005) and "federal law is silent on..." (superseded-003) are negative claims, supported by absence records and by the resolution rather than by a provision.conflict-011 asserts what LEAVE-001 says while citing only the Ohio absence record, so LEAVE-001 was never shown to the verifier.compose is the weak link, not verify. It emits no citation at all in six cases, and overclaims beyond its cited text in about eight more. The verifier is doing its job on both.
And verify's evidence set is too narrow. It should see the resolution and the passages the answer was told to address, not only the ones it relies on. Otherwise every precedence conclusion and every recorded silence is unverifiable no matter which model reads it.
verify's evidence to include the resolution and every retrieved passage, so derived conclusions and absence records can be checked at all.compose's two failure modes, emitting no citation and asserting beyond the cited passage. Both are visible in the trace and neither needs a model change.The general lesson, and it is the one this project keeps relearning: I had a plausible mechanism, a suggestive number (15 of 19), and a pre-registered plan built on both. Reading fourteen claims against fourteen passages took one cached run and no money, and it inverted the conclusion.
Status: implemented, Steps 2a and 2b. Not re-measured end to end (credits).
DL-34 found the verifier was mostly right and the failures were elsewhere. Reading the seven "decidable" rejections finished the correction.
I had recorded those seven as "compose emitted no citation". That was wrong too, and for an embarrassing reason: I read citations from the final state, which verify empties when it discards an answer. I was looking at the wreckage and describing it as the cause.
Five of them failed the figures check, and the flagged figure came from the question:
| scenario | flagged | where it came from |
|---|---|---|
| conflict-007 | 11 |
"I'm leaving with 11 unused vacation days" |
| conflict-018 | 11 |
the asker's tenure |
| straightforward-006 | 160 |
the asker's hours |
| ambiguous-015 | 9 |
the asker's own figure |
An answer repeating the asker's number back is not a fabrication. The check exists to catch figures invented about the law. It now treats the question and the supplied employee context as supporting evidence, and still flags a number that appears in neither.
Five flagged claims had their support withheld by construction, because cited_text was built only from passages the answer cited. Three kinds of claim could never be checked:
conflict-011 describes what LEAVE-001 says while citing only the Ohio absence record.verify now receives the resolution as explicit evidence, plus cited passages, absence records, and anything the answer was told to address. The prompt says which evidence supports which kind of claim.
Targeted rather than everything. Sending all twelve retrieved passages fit the free tier's ceiling with about two hundred tokens to spare, which is not headroom. The selection sends three, at ~1,000 tokens against a 6,900 budget.
Run against the open model, the superseded-003 claim "Ohio has no state paid sick leave statute" now returns supported, on 560 input tokens at $0.00.
That is one case, not a measurement. It does establish that verify runs comfortably on the free tier, which resolve never could, so re-measuring this does not depend on Anthropic credits.
The blocker was described as "verify rejects correct answers because it self-grades". Neither half survived. It was a false-positive figures check and an evidence set that made the system's own conclusions unverifiable. A cross-family verifier, the pre-registered fix, would have addressed neither.
PROMPT_VERSION is now verify-v2, so every cached verdict is reissued: the evidence and the instructions both changed, and reusing verdicts formed under the old conditions would compare two different questions.
Status: applied, Phase 10 verification. Ohio absences now 8 of 8.
DL-19 closed four of eight Ohio records and left four at the weaker "searched, not found, scope stated" standard. Closing them found a fifth error, in the same shape as the first three.
witness_duty_pay claimed: "Ohio has no statute requiring a private employer to pay an employee for time spent appearing as a witness under subpoena."
ORC 2945.451, read in full at codes.ohio.gov:
"This section generally does not require ... an employer to pay an employee for time lost as a result of attendance at any criminal proceeding. However, if an employee is subpoenaed to appear at a criminal proceeding and the proceeding pertains to an offense against the employer or an offense involving the employee during the course of his employment, the employer shall not decrease or withhold the employee's pay for any time lost as a result of compliance with the subpoena."
ORC 2151.211 says the same for juvenile and family proceedings. Both also forbid discharging or penalising an employee for attending.
The record was right about the general rule and wrong about the exception, which is DL-11's pattern exactly: a claim true enough in general and unsupported in the specific case that matters. And the case that matters is the one the eval asks about. control-008 is "Do I get paid if I'm called as a witness for the company?", which is the exception nearly verbatim.
And it was outside Title 41 again. The provisions sit in Chapter 2945 (criminal procedure) and Chapter 2151 (juvenile). DL-19's finding was that a single-title sweep clears these confidently; a fifth instance of the same thing is no longer a coincidence about one search, it is a property of the subject. Employment obligations in Ohio are scattered across the code by the context that creates them, not gathered under labour.
control-008 keeps company as its expected authority. The handbook pays for witness duty generally; the statute protects pay only for criminal and juvenile proceedings concerning an offence against the employer. The handbook is broader, so it still controls the question as asked, and following the statute into its sub-case would need facts the question does not supply. That is the chain DL-19 stopped following with Cal. Lab. Code 246.5, stopped again here for the same reason. The scenario note records the statute so the narrowing is visible.
DL-19 defined a negative finding as "searched, not found, scope stated", and there was nowhere to state the scope. A reader could not tell a record backed by a cross-title sweep from one backed by nothing, and neither could the agent.
scope_searched is now a field on every record, and it travels into source_note so the retrieved document carries it. The three records that were correct are now correct and legible as correct: each names the titles and chapters checked, and says explicitly that the leave provisions found (ORC 124.38, the 3349/3352/3358 institutional rules) bind public employers and impose nothing on private ones.
Both are marked "no scenario turns on it". Closing them means either research the eval does not fund, which DL-6's stopping rule forbids, or an assertion from recall, which DL-3 forbids. An open box with a stated reason is a stronger artifact than a closed one with a guess behind it.
Status: partial run, 49 of 92 scenarios. Aggregate deliberately unclaimed.
The first run with a cross-family verifier: Haiku composing from cache, gpt-oss-120b verifying live. 43 scenarios could not run.
| cause | n |
|---|---|
| Groq per-minute rate limit | 37 |
| gpt-oss did not call the tool | 2 |
| other 400s | 3 |
| Anthropic credits | 1 |
The run reports fully_correct_strict of 0.612. That number must not be set beside the 0.620 baseline. Only 49 scenarios scored, only 15 reached verify against 58 before, and the slices are badly skewed: straightforward fell from n=17 to n=2, conflict from 18 to 7.
Two figures over different populations are two different measurements. The partial-run machinery did its job. A result with a stated denominator beats an outage. A stated denominator is still a warning rather than a licence.
On the 49 scenarios scored in both runs:
| fully correct before | 31 / 49 |
| fully correct after | 32 / 49 |
| recovered | 2 |
| regressed | 1 |
Net +1 is inside the one-scenario noise floor DL-23 established, so the fixes cannot be claimed to have improved the aggregate. What can be said is which two recovered:
conflict-001, the spec's flagship handbook-conflict case, the one that degraded to a referral and forced the demo's conflict button to be swapped in DL-29.conflict-007, the "I'm leaving with 11 unused vacation days" case, which is precisely the figures false positive DL-35 fixed.Both fixes did the specific thing they were built to do, on a sample too small to say more. control-010 regressed and is unexplained.
gpt-oss-120b intermittently refuses to call the tool on the verify prompt, returning "Tool choice is required, but model did not call a tool". That is the exact 400 that disqualified gpt-oss-20b as a provider in DL-24.
Two of 43, so intermittent rather than fatal. But forced structured output is the contract every node in this system depends on, and a verifier that sometimes declines to answer cannot be deployed as one. DL-24 fixed the adoption threshold on accuracy; it did not anticipate a model failing the interface.
DL-24 pre-registered a cross-family verifier as the fix for verification, and two independent measurements now point away from it. DL-34 showed the verifier was mostly right and the defects were a false-positive figures check and a withheld evidence set. This run shows the open model cannot reliably hold the tool contract on that prompt.
The arm is worth reporting as a finding rather than adopting. That is a negative result on a pre-registered plan, which is the outcome the pre-registration existed to make reportable.
The run is resumable from cache and Groq's daily budget is a rolling 24-hour window, so the 43 skipped scenarios cost nothing tomorrow. The full-population comparison is deferred, not abandoned, and the aggregate stays unclaimed until it exists.
resolve, pre-registered before the runStatus: pre-registration. Written and committed before any capped arm was scored. The mechanism below is measured; the outcome is not.
resolve sends every retrieved passage. Nobody ever asked what it needs. That question was never posed because retrieval was tuned for recall and the prompt was whatever recall produced, which is how a context window becomes a default rather than a decision.
DL-15's rule is that a wrong mechanism invalidates the experiment that rests on it, so the mechanism was measured before a threshold was fixed. All figures below cost $0.00: retrieval runs on cached embeddings and resolve was replayed cache-only.
Across the 57 answer scenarios:
| layer | mean passages | share of passage tokens | absent in |
|---|---|---|---|
| federal | 6.11 | 54.2% | 0/57 |
| state | 2.28 | 35.9% | 14/57 |
| company | 1.74 | 9.8% | 4/57 |
Federal is the bulk, which is what makes a per-layer cap the right lever rather than a global truncation. A global cut of k would shrink the layers that are already thin, and precedence needs one finding per layer to compare at all.
Rank within its own layer of the passage resolve actually cited, over 88 citations:
| rank | n | cumulative |
|---|---|---|
| 1 | 74 | 84.1% |
| 2 | 9 | 94.3% |
| 3 | 2 | 96.6% |
| 4 | 2 | 98.9% |
| 5 | 1 | 100.0% |
84% of the evidence resolve uses is the top passage of its layer. The tail is small, and it is not randomly distributed: all three citations deeper than rank 3 are in the conflict slice (conflict-006 federal rank 5, conflict-011 federal rank 4, conflict-020 state rank 4). The slice the project exists to get right is the slice that reads deepest.
Groq's free tier is 8,000 tokens per request and charges reserved max_tokens, not tokens produced. resolve reserves 2,048, so the prompt competes for 5,952, not 8,000. Measuring against 8,000 would have understated the problem by half.
| cap | mean | p95 | max | over ceiling | cited passages dropped |
|---|---|---|---|---|---|
| none | 5,570 | 9,178 | 10,242 | 17/57 | 0 |
| 4 | 4,104 | 6,863 | 7,437 | 10/57 | 1 |
| 3 | 3,491 | 5,706 | 6,280 | 2/57 | 3 |
| 2 | 2,789 | 4,565 | 4,870 | 0/57 | 5 |
30% of uncapped prompts exceed the binding ceiling, which reconciles with the 23% recorded in CLAUDE.md under a different prompt version.
The obvious experiment is capped against uncapped on one model. It is not runnable and running it would have been worse than not.
Anthropic is out of credits, so the arm has to be free, and uncapped prompts fit the free ceiling in only 10 of 18 conflict scenarios. Those 10 are systematically the evidence-light ones: precisely the scenarios where a cap removes nothing. A paired comparison on that subset is structurally biased toward adopting the cap.
That is DL-24's bias with the sign flipped, and it was caught there by the same reasoning: "the excluded ones are systematically the evidence-heaviest, so scoring the remainder would flatter the open model." A result that can only come out one way is not a measurement.
cap=2 makes all 57 scenarios fit the free tier (conflict 18/18, control 10/10, straightforward 17/17, adversarial 2/2, superseded 10/10). cap=3 leaves 2 over. Both arms therefore run on every conflict scenario, and the comparison is unbiased in a way the uncapped one cannot be.
So the question is not "is capping safe" but the sharper one: where does trimming start to cost accuracy?
openai/gpt-oss-120b, free.run_precedence already supports only_slice for exactly this budget reason.If cap=2 and cap=3 score identically and both sit at or above the Haiku reference, the honest reading is not "capping is free". It is that n=18 cannot resolve a one-scenario difference, which DL-23 already established as this project's noise floor. That outcome gets reported as underpowered, not as a win.
Whether capped context matches uncapped context is unanswerable without a model that has both a wide context window and a free tier. It is deferred, not abandoned, and it is the one question a wide-context free provider would unblock.
Status: a statement of scope, not a plan. Written because a portfolio piece that quietly omits production concerns is indistinguishable from one whose author did not know about them.
Every item here was raised, costed, and left undone on purpose. Where something was done, it says what and how far.
Quality targets were pre-registered properly and honoured: 0.80 macro route accuracy as an upgrade trigger (DL-22), a 2-point tie-break for chunking (DL-14), a 10-point adoption bar for reranking (DL-16), a 2-scenario bar for a retrieval guarantee (DL-28). Each was fixed before the data existed, and each decided something.
No latency, throughput or availability target was ever written down. That is the gap, and naming it precisely matters more than closing it now would.
Latency was measured only reactively, in DL-28, and only because a filtering decision needed it: 5.8 ms median retrieval against a 13,198 ms end-to-end, 0.04% of the budget. The figure settled the question it was raised for and nothing else was asked of it.
Had a p95 target been fixed on day one, four sequential model calls would have been a design constraint rather than an observation. triage → resolve → compose → verify is a serial chain, and no amount of retrieval tuning moves it. That is the same shape as this project's central claim about relevance and correctness: the expensive thing is not the thing everyone tunes.
| target | current | what closing it would take |
|---|---|---|
| p95 latency | 13.2 s typical, never targeted | parallelise or drop a node; both change behaviour |
| throughput | single instance, never load-tested | no load test exists |
| availability | undefined | no fallback path when a provider is down |
Availability's absence was demonstrated rather than theorised. When the Anthropic credit balance reached zero the system did not degrade, it stopped. The pre-computed scenarios kept serving, but that is a property of static files, not a designed fallback.
There is no checkpointer. LangGraph ships one and this graph does not use it, so a request that fails at compose re-runs from triage.
That is defensible for a demo whose expensive path is rate-limited to single figures per day and indefensible for anything doing real volume.
What was fixed is the eval harness, and only after it lost data. DL-37 reports a run that reached 49 of 92 scenarios, losing 43 to a rolling daily cap, and could only report a paired subset as a result. The remedy was per-scenario failure tolerance with a stated denominator, then the pre-flight budget guard in eval/run_context_sweep.py that stops cleanly rather than raising on scenario N. Both are reactions to a loss, not a design.
The one durability property that worked was accidental. The on-disk model cache exists to stop re-runs costing money; it is also why a killed run resumes for free and why the Haiku control arm survived the credit outage at all. Durability as a side effect of a cost optimisation is luck, and it is recorded here as luck rather than as architecture.
api/limits.py holds its windows in process memory, and says so in its own module docstring. On the single Fly instance the spec ships, per-process counters are right. On two they silently double every limit, including the global spend breaker.
Left alone deliberately. The sentence is worth more than the implementation would be: a shared counter needs Redis, which is an external dependency and a new failure mode bolted onto a demo that currently fits in one container.
None exist. Langfuse provides observability, not alerting: nothing pages when the error rate moves, because nobody is on call.
TRACE_REDACT defaults to empty, so employee context reaches Langfuse. For a corpus of invented scenarios containing no real employees, redaction would hide the trace the demo exists to show.
It is documented in .env.example and in agent/tracing.py, and a deployment carrying real employee data should set it before anything else. That is a judgment about this deployment rather than a general one, and it is the only item here that would change on day one of real use.
The experiment has not been run. The pre-registration above stands as written; no arm was completed, and no number from it may be quoted.
The per-layer cap is built, tested and mutation-covered, and DEFAULT_PASSAGE_CAP is None, so resolve behaves exactly as it did before. Nothing that ships depends on this experiment, which is why it can be left open without qualifying any other result.
Two attempts were made, both on Groq's free tier. The pacer prices calls from estimate_tokens, which is characters divided by four, while the provider counts real tokens, so at roughly 5,300 charged tokens against an 8,000-per-minute bucket the run sustains about one call a minute and loses scenarios to 429s at the margin. Retrying recovered conflict-003 and conflict-005; conflict-016 and conflict-017 survived three 65-second retries and were still refused.
One finding does survive the pause, because it is not about the budget. conflict-006 and conflict-009 fail with BadRequestError on every attempt, throttled or not. That is the same tool-contract failure DL-37 recorded for gpt-oss-120b on verify prompts, now observed on resolve prompts as well. It is a property of the model, it will recur on a funded account, and it strengthens rather than weakens DL-37's conclusion: the open model cannot be relied on to hold forced structured output, which is the contract every node in this system depends on.
What it would take to finish. The full sweep is 36 calls at roughly 155,000 input tokens. On Haiku that is a few minutes and well under a dollar. The deferred capped-against-uncapped half needs the same thing, because uncapped prompts do not fit a free per-request ceiling at all. Both are blocked on the same trivially small amount of credit, and neither blocks the deployment.