Implementation plan

Contents

Phases 0 to 5: corpus, ground truth, retrieval

Goal: A public, traced agent that answers HR leave questions by determining which authority controls, with an eval harness and a decision log proving the design choices.

Architecture: Three-layer corpus (federal / state / company) in Qdrant with jurisdiction and effective-date metadata. LangGraph agent routes between clarify, retrieve, resolve, compose, verify and refuse. FastAPI service behind a rate-limited public demo.

Tech Stack: Python 3.12, FastAPI, Qdrant, LangGraph, Langfuse, pytest, Docker Compose (local), Fly.io (production).

Spec: docs/superpowers/specs/2026-08-25-controlling-authority-design.md

Source verification (done 2026-08-25)

Checked live before writing this plan. These are facts, not assumptions.

Source Status Notes
eCFR Verified Official API. GET /api/versioner/v1/full/{date}/title-29.xml?part=825 returned 350KB of structured XML. Title 29 current to 2026-08-21. Point-in-time by date works.
California Verified leginfo.legislature.ca.gov is server-rendered. Full text of Gov Code 12945.2 present in the HTTP response. Needs an HTML parser, no JS.
Ohio Verified codes.ohio.gov is server-rendered. Statute body present in raw HTML. Pages carry Effective: <date>, which feeds effective_from directly.
New York Verified Open Legislation API, key obtained. GET /api/3/laws/WKC?depth=2 returns a navigable tree. Paid Family Leave sits in Article A9, Disability Benefits. Nodes carry activeDate, which feeds effective_from. docType/locationId map to section_path.

Ohio has no state family-leave statute for private employers. That is why it is the control case. The absence must be encoded explicitly (Task 3.3) or it cannot be tested.

PHASE 0: Foundations

Task 0.1: Repo and skeleton

Files: Create pyproject.toml, .gitignore, .env.example, README.md, docker-compose.yml

cd job-search/dayforce-ai-engineer/controlling-authority
git init && git branch -M main

ANTHROPIC_API_KEY=
OPENAI_API_KEY=
VOYAGE_API_KEY=
NY_SENATE_API_KEY=
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=https://cloud.langfuse.com

Never commit .env. .gitignore must contain .env, __pycache__/, .venv/, corpus/raw/.

git check-ignore -v .env && echo "ignored, good"

git add pyproject.toml .gitignore .env.example README.md docker-compose.yml
git commit -m "chore: project skeleton"

Task 0.2: Register for the New York API key

python -c "import os,urllib.request;k=os.environ['NY_SENATE_API_KEY'];print(urllib.request.urlopen(f'https://legislation.nysenate.gov/api/3/laws/WKC?key={k}').status)"

Expected: 200. If this blocks, drop NY and run with CA and OH. Two contrasting states still make jurisdiction filtering load-bearing.

PHASE 1: Handbook and scenarios FIRST

This phase comes before any retrieval code. The scenario set is the largest piece of human effort, cannot be generated without destroying its own ground truth, and tells ingestion when to stop. Building the pipeline first is how projects like this end up with twelve smoke tests instead of an eval.

Task 1.1: Author the company handbook

Files: Create corpus/handbook/*.md

---
policy_id: LEAVE-004
title: Parental Leave
effective_from: 2025-01-01
effective_to: null
supersedes: null
---
  1. Below California minimum (statute must override)
  2. Above statute (handbook must control)
  3. Superseded, both versions retained with dates
  4. Topic absent entirely (forces refusal)
  5. Ambiguous applicability (forces clarification)

Record which policy carries which defect in corpus/handbook/DEFECTS.md. That file is ground truth and must never be fed to the agent.

Task 1.2: Scenario schema and the first slice

Files: Create eval/scenarios/schema.py, eval/scenarios/straightforward.yaml, tests/test_scenario_schema.py

def test_scenario_requires_expected_route():
    with pytest.raises(ValidationError):
        Scenario(scenario_id="s1", question="q", as_of_date=date(2026,1,1))

Task 1.3: The remaining scenario slices

Files: eval/scenarios/{ambiguous,conflict,superseded,out_of_scope,adversarial}.yaml

def test_scenario_slices_are_balanced():
    counts = Counter(s.expected_route for s in load_all())
    assert counts["clarify"] > 0 and counts["refuse"] > 0
    assert counts["answer"] >= counts["clarify"]  # or the agent learns to always ask

PHASE 2: Federal ingestion

Task 2.1: eCFR adapter

Files: Create ingest/federal_ecfr.py, tests/test_federal_ecfr.py

def test_parses_section_hierarchy():
    sections = parse_ecfr_xml(FIXTURE)
    s = next(x for x in sections if x.doc_id == "us:29-cfr-825.200")
    assert s.section_path == ["Part 825", "Subpart B"]  # ancestors only
    assert s.authority_layer == "federal"
    assert s.jurisdiction == "US"
    assert s.citation == "29 CFR 825.200"

curl -s "https://www.ecfr.gov/api/versioner/v1/full/2026-08-01/title-29.xml?part=825" \
  > tests/fixtures/ecfr_825.xml

Task 2.2: Point-in-time pull

def test_snapshot_date_is_recorded():
    recs = fetch_part(825, as_of=date(2020,1,1))
    assert all(r.effective_from <= date(2020,1,1) for r in recs)

PHASE 3: State ingestion

Each state gets its own adapter. Heterogeneous ingestion is the honest reality of forward-deployed work.

Task 3.1: California

Files: Create ingest/state_ca.py, tests/test_state_ca.py

Task 3.2: New York

Files: Create ingest/state_ny.py, tests/test_state_ny.py

Task 3.3: Ohio, and encoding absence

Files: Create ingest/state_oh.py, corpus/absence/oh.yaml

Ohio has no state family-leave statute for private employers. A retrieval miss and a genuine absence must not look the same to the agent:

- jurisdiction: OH
  topic: family_medical_leave
  finding: no_state_provision
  effect: federal_controls
  verified_on: 2026-08-25
  note: >
    Ohio has no state FMLA equivalent for private employers.
    Absence is a fact about the corpus, not a retrieval failure.

def test_absence_is_not_a_retrieval_miss():
    r = lookup_state_provision("OH", "family_medical_leave")
    assert r.finding == "no_state_provision"
    assert r is not None  # absence is a record, never None

PHASE 4: Chunking, embedding, vector store

Task 4.1: Structure-aware chunking

Files: Create retrieval/chunking.py, tests/test_chunking.py

Task 4.2: Embedding, as decision log entry #1

Files: Create retrieval/embed.py, eval/decision_log.md

### D1: Which embedding model for a regulatory corpus?
**Hypothesis.** A legal-domain embedding model beats a general-purpose one on
retrieval recall for statutory text, because the corpus is dense with terms of art.
**Metric.** recall@10 on the full scenario set.
**Result.** _pending Task 5.3_

Task 4.3: Qdrant collection

Files: Create retrieval/store.py, tests/test_store.py

def test_jurisdiction_filter_excludes_other_states():
    hits = search("parental leave", jurisdiction="OH", k=20)
    assert all(h.payload["jurisdiction"] in ("OH", "US") for h in hits)

PHASE 5: Baseline and eval harness

The baseline is built before the agent. Without a measured baseline the decision log has nothing to compare against, and the demo's side-by-side toggle has nothing to show.

Task 5.1: Naive RAG baseline

Files: Create agent/baseline.py

Task 5.2: Eval harness

Files: Create eval/run.py, eval/metrics/*.py

Task 5.3: Measure, and close D1

PHASE 6 onward: outline

Phases 0 to 5 are specified at task granularity because they are the critical path to a measured baseline. The phases below are scoped but deliberately not expanded into steps yet, because their design should be informed by what Phase 5 measures. Expand each when reached.

Phase 6: Agent graph. LangGraph nodes per the spec. One task per node, each with scenario-slice tests. resolve implements the four precedence rules and emits a structured trace. verify uses a different model family than compose.

Phase 7: Observability. Langfuse spans per node, recording retrieval filters, precedence decision, per-stage cost and latency.

Phase 8: API and protection. FastAPI. Per-IP limit, per-session daily quota, global daily circuit breaker, input length cap, pre-computed responses for the six curated scenarios.

Phase 9: Demo UI. Six scenario buttons, baseline toggle, live trace panel, free text input. The supersession scenario runs one question at two dates side by side.

Phase 10: Ship. Qdrant Cloud, Langfuse Cloud, app on Fly.io. README leads with the decision log, setup last.

Risks

Phases 6 to 10: the agent, and shipping it

Goal: Close the conflict-slice gap that retrieval cannot, then ship it publicly with traces and an honest decision log.

Written after Phase 5, deliberately. The original plan left these phases as outlines because their design should follow the numbers. It does.

What Phase 5 actually established

finding consequence for this plan
Best retrieval: 0.895 recall@10 with voyage-law-2 + structure-aware Retrieval is not the bottleneck. Do not tune it further.
Four of five slices at or near 1.000 The agent must not regress them while fixing the fifth.
Conflict slice: 0.722 recall@10, 0.556 recall@3 The whole of Phase 6.
On conflict scenarios, the handbook ranks first Precedence must override rank, not follow it.
Reranking headroom 7.0 pts, below threshold Do not build a reranker.
Numbers are oracle-filter Query rewriting is on the critical path, not a nicety.

The single most important fact: retrieval already returns the right document in the conflict slice 72% of the time. The failure is not finding the statute, it is knowing the statute beats the handbook that outranked it.

PHASE 6: The agent

Six nodes. Each task ends with the scenario set run against it, because a node that improves one slice while breaking another is not an improvement.

Task 6.1: Graph skeleton and state

Files: Create agent/state.py, agent/graph.py, tests/test_agent_state.py

Carries everything a later node needs, so nodes never re-derive:

class AgentState(TypedDict):
    question: str
    employee_context: EmployeeContext   # may be partially empty
    as_of: date
    route: Route | None                 # answer | clarify | refuse | escalate
    missing_fact: MissingFact | None
    retrieved: list[SearchHit]
    resolution: Resolution | None       # which layer controls, and why
    answer: str | None
    citations: list[str]
    verification: VerificationResult | None
    trace: list[TraceEvent]             # every node appends; never overwritten

Task 6.2: triage, which is also query rewriting

Files: Create agent/nodes/triage.py, tests/test_triage.py

DL-16 committed query rewriting here. It is not optional: two hard filters cannot be applied without it.

Task 6.3: clarify, and not over-clarifying

Files: Create agent/nodes/clarify.py

Task 6.4: resolve, the core

Files: Create agent/nodes/resolve.py, agent/precedence.py, tests/test_precedence.py

This is the phase. Everything else is scaffolding around it.

Rules 1 to 5 from the spec are deterministic given a set of retrieved provisions and their layers. A statutory floor comparison is arithmetic. Supersession is a date comparison. Silence is a set membership test.

Only two things genuinely need a model: deciding whether a provision speaks to the question at all, and comparing which of two provisions is more generous where that is not numeric. Everything else is a function.

This is the same reasoning that made verify partly deterministic: code cannot share a blind spot with a model, and it cannot be argued into a wrong answer.

class Resolution(TypedDict):
    controlling: Authority              # federal | state | company
    reason: PrecedenceRule              # which of the five rules decided it
    considered: list[LayerFinding]      # every layer, what it said
    non_controlling_to_address: list[str]

Task 6.5: compose

Files: Create agent/nodes/compose.py

Task 6.6: verify, mostly deterministic

Files: Create agent/nodes/verify.py

Replaces DL-15's cross-family rule. Code cannot share a blind spot with the model that wrote the answer, because it is not reasoning.

Task 6.7: End-to-end scoring

PHASE 7: Observability

Langfuse spans per node: retrieval filters and hits, the precedence decision and which rule fired, per-stage token cost and latency, final route.

The trace is surfaced in the UI, not hidden behind a debug flag. Explaining an AI system to non-technical stakeholders is the job this project is auditioning for; a visible reasoning trace demonstrates that rather than claiming it.

PHASE 8: API and protection

FastAPI. Public inference is exposed spend, so the protection mirrors what already runs in production on the Primrose & Eve workers: per-IP rate limit, per-session daily quota, global daily circuit breaker sized so the worst case bounds spend rather than draining a budget, hard input cap, and pre-computed responses for the curated scenarios so the path most reviewers take costs nothing and returns instantly.

PHASE 9: Demo

Six scenario buttons, a baseline toggle, a live trace panel, free text input, and one scenario that runs the same question at two dates to show supersession.

The baseline toggle is the single highest-leverage feature. Same question, naive RAG beside the full agent. On a handbook-conflict case the baseline confidently returns the handbook's wrong answer and the agent catches that statute overrides it. That delta is the entire argument, and it lands in one screen with no explanation.

PHASE 10: Ship

Qdrant Cloud, Langfuse Cloud, single container on Fly.io. README leads with the decision log; setup last.

Risks