Covers GAP-11 (INV-RET4/G8) and GAP-14 (INV-QA1/G10). Retrieval quality was
never measured (only telemetry observation) and the halacha review backlog was
invisible (the 10/19 gap was found by accident).
Unit B — backlog visibility (pure code, container):
- metrics.halacha_backlog(conn) → {pending_review, approved, rejected, published,
total, oldest_pending_at}; surfaced in metrics.get_dashboard() (get_metrics MCP
tool) and /api/system/diagnostics. Live count revealed 178 pending / 1552 total,
oldest from 2026-05-03 — previously invisible.
Unit A — retrieval eval harness (host-side scripts):
- scripts/eval_gold_bootstrap.py — seeds data/eval/gold-set.jsonl. Two sources:
citations (cited==relevant via search_relevance_feedback — empty until decisions
cite precedents) and known_item (query=case_name → relevant=self; a real
citation-free signal, the methodology #52 checked by hand). Idempotent; preserves
source='chair' rows.
- scripts/eval_retrieval.py — runs the production retrieval path (search_library /
search_internal) over the gold-set; computes precision@k, recall@k, MRR, nDCG@k
(k=5,10); aggregates overall + per-corpus + per-practice_area; writes a report and
a delta vs committed baseline.json (which records the retrieval_config it reflects).
--self-test unit-checks the metric math offline.
Gold-set strategy = hybrid (chair decision): bootstrap + chair review. The citation
source is empty today (0 cited precedents in decisions), so the seed is known-item
(77 queries: 54 internal_decisions + 23 precedent_library). The gold-set is
PROVISIONAL until Dafna reviews it (the domain chair-gate).
Baseline (production config: multimodal+rerank on): R@10=0.987, MRR=0.837,
nDCG@10=0.872. Finding: MULTIMODAL_ENABLED=true slightly lowers known-item recall
(image-page results displace exact name matches) — relevant to #15. precedent_library
weaker than internal (R@10 0.957 vs 1.0) — one external precedent unfindable by name.
"CI gate" realized as discipline (re-runnable harness + committed baseline + run
before/after any retrieval-layer change) — retrieval needs prod DB + Voyage, no CI
runner has that access.
Spec: docs/superpowers/specs/2026-05-31-fu5-eval-harness-design.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deterministic migration of ~52 internal_committee rows whose case_number holds
a full citation → normalized bare number (citation_formatted already correct).
DB analysis (2026-05-31): clean 1-token extraction, 0 key-collisions, 0
citation↔case_number mismatches, no month-padding dups. Chair-gated reversible
migration (backup→dry-run→approve→apply). One edge for chair: 8047/23 ערר vs בל"מ.
External (#68/FU-2c) split out — its citation_formatted is inconsistent.
Verified all 11 case_law FKs use id(UUID), not case_number → rename is FK-safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
content_hash/indexed_hash change detection + reindex_case_law from stored
full_text (no re-OCR) + drift health-check. Verified vs 3+ sources (content-
hash change detection, RAG re-embed-on-edit). #61.2 multimodal backfill closed:
42 rows are text-ingested (document_id NULL, no source PDF) — page-images
impossible without a PDF to render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Establishes the foundation to fix a recurring root-cause failure class
(non-canonical identifiers, asymmetric ingest paths, silent manual gates):
- Confirmed system mission (quasi-judicial decision assistant; human decides)
- Decomposition into 5 sub-projects (spec → audit → integrity layer → re-check → process agents)
- spec-set structure under docs/spec/ (lifecycle-organized + cross-cutting files)
- 11 global invariants + engineering rules, each backed by ≥3 authoritative sources
(NCSC/JTC, FJC, CEPEJ, South Bucks; RAG/Lewis, Manning IR, Elastic/Pinecone/Weaviate;
DAMA-DMBOK, ISO 8000, ISO 15489, Kleppmann, Codd, Fowler)
- 3-source verification protocol; UNVERIFIED items escalated, not decided solo
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settings page extension to view and edit MCP server config (env vars,
tools, client registrations) — hybrid edit model: non-secrets editable
through Infisical, secrets read-only with drift detection vs container.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
