Controlling Authority

Ask how much leave you get and three authorities might answer: federal law, your state, and your company handbook. The handbook is the closest match to the question, and when it promises less than the law requires, the closest match is wrong. Nothing in the text tells you which one wins.

I picked this because it is a real problem in HR software, and because better retrieval does not solve it. Fetching the right passage is the easy half. Something still has to decide which authority governs, and I wanted to know whether that decision belongs in code instead of a prompt.

It does. Same retrieval, same runs, one component swapped, 22.8 points apart. The decision log has the rest, including the experiments that failed and the features I built, measured, and dropped.

How to read this

Pick a scenario below. Each is a recorded run against the real corpus, instead of a canned response.
Read the two cards side by side. The left card is this system, which applies precedence rules in code. The right card is what a system that trusts the closest matching passage concludes, from the same retrieval, on the same run.
Look at the tag on each card. Violet means the run matched the labelled ground truth. Red means the answer was wrong.
Open How it decided for the steps the agent took, and open any step for the raw detail behind it.
Read the score under the trace. That is how the whole slice scores, not how this one case went.

Start with conflict. The handbook is the closest match to that question and the wrong answer, so the two cards disagree.

Example scenarios

Each one is a recorded run, instead of a canned response. They cost nothing to serve, so no rate limit applies to them.

straightforward: federal law answers it, and both systems agree
ambiguous: the correct output is a question, not an answer
conflict: the handbook beats the statute, and the naive system picks the wrong layer
superseded: one layer speaks and the others are silent, so silence grants nothing
refuse: outside leave entirely, so the system declines rather than guessing
escalate: a person needs to handle this, not a policy lookup
point in time: the same question on two dates, answered from the law in force on each

Ask your own

Rate limited, because live inference costs money. Change the date to see which law was in force then.

Engineering choices

hybrid search: dense plus sparse vectors, fused with RRF, because statutory text is full of exact terms a dense embedding blurs
filters: jurisdiction and date are hard constraints inside each search, never ranking signals
chunking: structure-aware, picked over fixed-size by measurement
reranking: not built. The rule said build one only above 10 points of headroom, and it measured 7.0. Reopened in DL-40
latency: retrieval is 0.04% of the budget. The cost is four sequential model calls, so these scenarios are pre-computed
tokens: 10 passages retrieved, not 30. Verification sees 1,000 tokens of targeted evidence, not 6,700
guardrails: forced tool calls, citations checked against what was retrieved, spend caps, and the answer key kept out of the corpus
hallucination: four of five groundedness checks are code, not a second model grading the first
sampling: temperature and top-p are unset, which is a gap rather than a choice. DL-41
drift: records carry the prompt and corpus versions they ran under and are marked stale when those move
knowledge graph: not used. Three layers with a precedence order is a rule set, not a graph

The README has the reasoning for each, and the decision log has the measurements.

Where this falls short

When it should ask you a follow-up instead of answering, it answers anyway 47% of the time.

The answer skips your handbook three times out of four when the law overrides it. You already read that policy, so being told it no longer applies is the part you need.

A third of answers get swapped for a referral, and the citations go with them. What I tried against each is in the decision log.