Tagging is easier one source-type at a time. goldset_list now returns
case_law.source_type; the page adds:
- a filter (הכל / פסקי דין / ועדת ערר) with live counts,
- a group-sort so even in "הכל" all court rulings come first, then all
committee decisions,
- a per-card source badge (פסק-דין / ועדת ערר).
Verified: tsc --noEmit 0; source_type splits the live batch 58 court / 92 committee.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the CSV-edit workflow with an in-app tagging page so the chair/Dafna
can label the extraction-quality gold-set by clicking, and see validator
precision/recall live.
Schema (V29): halacha_goldset — a stratified, human-tagged evaluation batch
(is_holding / correct_type / quote_complete, NULL until tagged).
db.py:
- goldset_create_sample (stratified round-robin over case×rule_type, idempotent),
- goldset_list (items + halacha content + the machine's own labels),
- goldset_tag (partial — one field at a time for keyboard tagging),
- goldset_score (ports the script's P/R/F1: each validator scored as a
not-a-holding detector against the human tags — the #81.8 input).
API: GET /api/goldset, POST /api/goldset/sample, GET /api/goldset/score,
PATCH /api/goldset/{id}.
web-ui:
- lib/api/goldset.ts (hooks),
- components/goldset/goldset-panel.tsx — card-per-item, keyboard-first
(J/K nav, H/N holding, C/X quote), progress bar, hide-tagged toggle, and a
collapsible live score table,
- app/goldset/page.tsx + nav link "מדגם-זהב" under ידע ולמידה.
Methodology guard kept explicit in UI + docstrings: tags are HUMAN ground truth,
no AI pre-fill (circular bias). Populated a 150-item stratified batch.
Verified: backend create/list/tag/score against the live DB; tsc --noEmit 0;
py_compile ok. (Local Turbopack build blocked by worktree symlink — CI builds clean.)
Invariants: G1 (eval set modeled at source in its own table); G2 (reuses the same
halacha_quality validators the extractor runs — no parallel scoring logic).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-precedent recurrence of a principle is real but is NOT citation
corroboration (X11) — the 5 candidate pairs have ZERO citations between their
precedents. Recording them in halacha_citation_corroboration would fabricate
citation data and inflate corroboration_count. This adds a proper, separate
halacha-level link for parallel authority.
Schema (V28): equivalent_halachot — symmetric (halacha_a < halacha_b, CHECK +
UNIQUE), non-citation, cross-precedent-only. ON DELETE CASCADE.
db.py:
- link_equivalent_halachot (idempotent; rejects same-id and SAME-precedent pairs
— parallel authority is cross-precedent by definition), unlink, and
list_equivalent_for_halacha.
- list_halachot gains include_equivalents → _annotate_equivalents attaches an
`equivalents` list (both directions) per row.
API: include_equivalents on GET /api/halachot; GET/POST/DELETE
/api/halachot/{id}/equivalents for the chair to view/link/unlink manually.
scripts/halacha_batch_reconcile.py: --link records found cross-precedent pairs
as equivalent_halachot (non-destructive, idempotent).
web-ui: Halacha.equivalents type; the clean review queue fetches
include_equivalents; the review card shows a gold "עיקרון מקביל ב-N" badge + an
expandable list (case + rule + similarity) labeled "אסמכתה מקבילה — לא ציטוט".
Populated the 5 reviewed pairs (chair decision: keep all + link as parallel
authority). Verified: 5 rows; the 1023-20 hub annotates 3 of its halachot with
equivalents; tsc --noEmit exits 0.
Invariants: G1 (model recurrence at source in its own table, not by abusing the
citator); G2 (no parallel path — extends list_halachot); citator integrity
preserved (corroboration stays citation-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes #84 — surfaces the backend gating/prioritization (#84.1/#84.3, PR
#93) in the chair's review UI and adds near-duplicate clustering (#84.2).
Backend
- db.list_halachot gains `cluster` (#84.2): annotates each row with cluster_id +
cluster_size by unioning same-precedent halachot within HALACHA_CLUSTER_COSINE
(0.90, new config). Display-only — never merges/deletes. Pairwise is confined
to the returned set (cheap).
- GET /api/halachot exposes the `cluster` query param (default off).
Frontend (web-ui)
- Halacha type gains optional cluster_id / cluster_size (hand-written module; no
api:types regen needed — halachot aren't typed off the generated schema).
- useHalachotPending(opts): the default "clean" queue now fetches
exclude_low_quality + order_by_priority + cluster; needsFix:true returns the
flagged 'needs extraction fix' bucket (filtered client-side).
- HalachaReviewPanel: a "תור נקי / דורש תיקון-חילוץ" toggle (#84.1); near-dup
clusters collapse into ONE card showing "+N וריאנטים" with an expandable list,
and approve/reject/defer on a clustered card applies to all variants via the
batch endpoint (#84.2 + #84.4). Counts show true halacha totals (pendingTotal).
New flag labels added (application / near_duplicate / nevo_preamble_leak).
Verified:
- backend: list_halachot(cluster=True) on the live queue — algorithm correct
(groups related same-precedent rules at 0.78; none at the production 0.90
because dedup #82 already removed near-dups — the desired state).
- frontend: `tsc --noEmit` exits 0 (type-clean); no new lint errors (the one
lint error is pre-existing in training/learning-panel.tsx from #94). Local
Turbopack build can't run on the worktree node_modules symlink — CI builds in
a clean checkout.
Invariants: G1 (gate/cluster at source in SQL, not post-hoc); G2 (same
list_halachot path); §6 (flagged items routed to a visible bucket, not dropped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend for the halacha approval-queue triage (#84). The keyboard UI, batch
actions and defer/reject (#84.4–6) already shipped; this adds the gating,
prioritization and metrics the queue was missing.
db.list_halachot — two opt-in triage controls:
* exclude_low_quality (#84.1): drop items carrying ANY quality_flag
(application / quote_unverified / truncated / non_decision / thin /
nli_unsupported / near_duplicate) — they belong in a 'needs extraction fix'
bucket, not the chair's approve queue.
* order_by_priority (#84.3): active-learning order — negatively-treated
first, then most-uncertain (lowest confidence), then oldest — instead of
FIFO, so the highest-value decisions surface first.
halachot_pending (MCP) — now gated + prioritized BY DEFAULT; include_low_quality=
true reveals the needs-fix bucket. The agent review path benefits immediately.
GET /api/halachot — same two params, default OFF (non-breaking; the UI opts in).
metrics.halacha_backlog (#84.7) — splits pending into clean vs flagged, adds
deferred, reviewed_total, approve_ratio, and a pending_by_flag breakdown, so the
backlog distinguishes real review work from extraction noise.
Deferred (documented): #84.2 near-duplicate cluster cards and wiring the UI
fetch to the new params require frontend work + an api:types regen AFTER this
deploys (the new query params aren't in prod's OpenAPI until then) — a clean
follow-up. The backend fully supports both now.
Verified against the live DB (read-only):
- pending 177 → gated-clean 110, 0 flagged items leak into the clean queue.
- priority order surfaces the lowest-confidence items first (0.55, 0.55, ...).
- backlog: pending_clean=110 / pending_flagged=67 / approve_ratio=0.916,
pending_by_flag={nli_unsupported:59, quote_unverified:3, thin:3, truncated:2}.
- pytest tests/test_halacha_quality.py — 52 passed (no regression).
Invariants: G1 (gate at source — SQL filter, not post-hoc); G2 (no parallel
path — same list_halachot); §6 (flagged items routed to a bucket, never dropped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Halacha-extraction quality (#81) and dedup-on-insert (#82) — engine changes
(pure + tested) plus measurement/ops tooling.
halacha_quality.py
- #81.4 application gate: is_fact_dependent() (high-precision "applied to THIS
case" deixis per the strict rubric §3/§27) + FLAG_APPLICATION. compute_quality_flags
now takes rule_type and flags rule_type=='application' OR fact-dependent —
blocking auto-approve (an illustration is not a generalizable holding).
- #82.3 lexical tail signal: jaccard_shingles / normalized_levenshtein /
lexical_near_duplicate + FLAG_NEAR_DUPLICATE, for the 0.83–0.93 cosine band.
halacha_extractor.py — pass rule_type to the flag computation; re-type a
binding-labeled fact-application to 'application' (mirrors non_decision→obiter).
db.py (store_halachot_for_chunk) — dedup now fetches the nearest same-precedent
neighbor once: cosine ≥ DEDUP → skip (unchanged); cosine in [BAND, DEDUP) with
high lexical overlap → FLAG_NEAR_DUPLICATE (review, not skip — never drop a
possibly-distinct principle unreviewed).
config.py — HALACHA_DEDUP_BAND_COSINE (0.83).
Scripts:
- scripts/halacha_goldset.py (#81.7) — export stratified sample for human
tagging; score validators (P/R/F1) against the tags. Backbone for #81.8.
- scripts/halacha_batch_reconcile.py (#82.7) — conservative cross-precedent
dedup (cosine ≥0.95), dry-run report only.
- scripts/calibrate_halacha_dedup.py (#82.1) — calibrate the lexical thresholds
against the 2026-06-03 cleanup gold-set.
Deferred (documented): #82.4 merge-provenance and #82.5 DB ON CONFLICT/UNIQUE
on normalized quote are NOT included — the current skip+flag behavior is safe,
whereas a UNIQUE on normalized_quote would fail on existing dups and a blind
merge risks losing provenance; they need their own chair-reviewed migration.
#82.6 over-merge guard is moot until merge lands. #81.6 full rhetorical-role
classifier deferred (section pre-filter + application flag cover the practical
case); #81.8 blocked on the human-tagged gold-set (harness now provided).
Verified:
- pytest tests/test_halacha_quality.py — 52 passed (14 new).
- calibrate: configured (0.55,0.70) → precision 1.0 (zero false-merge), recall
0.30 — correct profile for an auto-approve-blocking signal.
- goldset export: 15-row sample CSV. batch reconcile: 819 halachot → 5
cross-precedent candidate pairs.
Invariants: G1 (normalize at source — flag at insert, not at read); §6 (no
silent swallow — suspect items flagged to review, never dropped); G2 (no
parallel path — same store_halachot_for_chunk / compute_quality_flags).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#86.2 backfill + #86.3 benchmark, plus a #86.1 over-strip fix found en route.
extractor.py
- extract_nevo_ratio(): capture Nevo's מיני-רציו block (editorial holdings
summary) before it is stripped — a free professional gold-set (#86.3).
- _DECISION_START hardening (#86.2): the merged #86.1 regex over-stripped.
(a) פסק-דין headers are markdown-wrapped (**פסק דין**); the old anchor
required the keyword as the first line char with one separator, so it
missed the header and matched a citation 32K deep (עמ"נ 50567-07-21,
losing 45% of the body). Now tolerates leading markdown + 0-3 seps,
and the final-nun form (דין ן vs דינו נ).
(b) bare השופט/הנשיא matched CITATIONS ("השופט מ' חשין, פסקה 23"). The
authoring-judge line ends with a colon; we now require it.
ingest.py
- capture the ratio before stripping and store it on the row (best-effort,
non-fatal); also strip the text-upload path (was file-only).
db.py
- add case_law.nevo_ratio column (additive); allow it in update_case_law.
scripts/backfill_nevo_preamble.py (#86.2) — dry-run-by-default data migration:
finds historically-leaked rulings, captures ratio→nevo_ratio, rewrites
full_text (+content_hash), reindexes, and FLAGS (never deletes) halachot whose
quote lives in the removed preamble (review_status=pending_review +
nevo_preamble_leak flag). Safety guard: rows with keep%<--min-keep (60) are
excluded from --apply as suspected over-strip. --apply writes backup+manifest
to data/audit/ first. Chair-gated — NOT applied here.
scripts/nevo_ratio_benchmark.py (#86.3) — LLM-as-judge (local claude_session,
zero cost) measures recall/precision/granularity of our halachot vs the Nevo
ratio. Works pre- and post-backfill (reads nevo_ratio, falls back to full_text).
Verified:
- pytest tests/test_nevo_preamble.py — 12 passed (incl. citation/markdown
over-strip regressions).
- backfill dry-run: 19 leaked rulings, 27 contaminated halachot, all ≥75%
keep (the 32K over-strip is gone).
- benchmark on בג"ץ 1764/05: recall=0.875 precision=1.0 granularity=1.75x.
Invariants: G1 (normalize at source — strip/capture at ingest, not at read);
no silent swallow (contaminated halachot flagged + reported, not dropped);
data-migration is dry-run-default with backup+manifest, chair-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
asyncpg עם pgvector register_vector מקבל את ה-embedding כ-list[float] ישירות;
str() גרם ל-DataError. תוקן בהתאם לדפוס store_*_image_embeddings.
Backfill הורץ בהצלחה: 2670 דוגמאות מ-83 החלטות.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
סוגר את לולאת-הלמידה (INV-LRN4): כל החלטה נסגרת מול הסופי, וכל סופי
מנותח מול הטיוטה. מזין את הטבלאות ש-T15 כבר קורא מהן.
T5 — פנקס-התאמה:
- SCHEMA_V26: טבלת draft_final_pairs (snapshot draft + final + diff + analysis + status).
- db: create/update/list_draft_final_pairs.
- mark-final (app.py): תופס snapshot של הטיוטה (decision_blocks) ברגע החתימה,
לפני שאפשר לדרוס אותו, ופותח שורת-פנקס (status=final_received).
T4 — דיסטילציה אוטומטית:
- learning_loop.process_final_version: משתמש ב-snapshot (לא בבלוקים שאולי השתנו),
מסווג style_method↔substance, שומר הצעה ב-pair (status=analyzed).
**הוסר ה-auto-upsert של style_patterns** — ביטל את ה-bug שדרס את שער-היו"ר
וזיהם סגנון במהות (INV-LRN1 + INV-LRN5).
- LESSONS_PROMPT: הפרדת style_method↔substance מפורשת + לקח מופשט בלבד.
- curator wake + hermes-curator.md: מריץ ingest_final_version ראשון; מציע רק
style_method שלא תועד; substance→מסלול precedent.
INV-LRN1 (שער-יו"ר, אין auto-commit) · INV-LRN4 (ניגוד-אמת) · INV-LRN5 (טוהר).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
עונה ל"להתחשב במה שכבר למדנו": הכותב התעלם מעריכות היו"ר ב-/methodology
(נשמרו ב-appeal_type_rules אך block_writer קרא רק קבועי lessons.py) ומ-
decision_lessons של /training. עכשיו הכל מגיע לכתיבה.
- db.get_methodology_overrides(category) — overrides של היו"ר (יחסי-זהב,
כללי-דיון, צ׳קליסטים) מ-appeal_type_rules (כמו merge של ה-API).
- db.get_recent_decision_lessons(limit, practice_area) — לקחי /training.
- _build_style_context(practice_area): מוסיף סעיף "⭐ למידה מצטברת — גובר
על ברירת-מחדל" עם שניהם, אחרי voice-fingerprint (T0). שני ה-callers מעבירים
practice_area. עובד יחד עם הלולאה (T4/T5) שתזין לאותן טבלאות.
תיקון-מספור (חלק מ-T9, דחוף כי T0 הזריק את הטעות): voice-fingerprint §3.1
תוקן — ההחלטה ממוספרת תמיד (מספור-אוטומטי ב-Word); "ללא מספור" היה
ארטיפקט-חילוץ. האנטי-דפוס האמיתי: רשימת-מיני בתוך פסקה + מספרים ידניים.
INV-LRN4 (הזרמת למידה) · INV-LRN5 (טוהר). G11.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
שתי בעיות UX בדף /precedents:
1. חילוץ מטא-דאטה לא נתן שום אינדיקציה שהוא רץ. בניגוד לחילוץ טקסט/הלכות
(extraction_status / halacha_extraction_status) למטא-דאטה היתה רק חותמת-זמן
metadata_extraction_requested_at — אין מצב "processing", לכן StatusPill לא
הציג כלום. נוספה עמודת metadata_extraction_status ('pending'|'processing'|
'completed'|'failed') במתכונת העמודות הקיימות, וה-worker
(process_pending_extractions + reextract_metadata) מעדכן אותה: processing
בתחילת פריט, completed בסיום (מנקה גם את החותמת), pending בכשל (לריטריי).
ה-UI מציג תג "מחלץ מטא-דאטה" + באנר מונה-אצווה עם אחוז התקדמות (high-water-mark
של עומק-התור) שמתעדכן אוטומטית דרך ה-polling הקיים (5ש').
2. שתי טבלאות מוערמות (בתי משפט / ועדות ערר) חייבו גלילה ארוכה. הוחלפו במתג-
מקטעים — טבלה אחת בכל פעם, עם שמירה על העמודות הייעודיות לכל סוג.
Invariants: G2 (מרחיב מנגנון-סטטוס קיים, לא מסלול מקביל), INV-TOOL4/GAP-45
(המשך חשיפת תור-החילוץ הסמוי). אין נגיעה בתוכן משפטי (G11).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INV-TOOL3 (idempotency על מפתח דטרמיניסטי). כל שלושת הכלים מחזירים את הרשומה
הקיימת במקום ליצור כפילות:
- case_create — מפתח case_number (כבר UNIQUE ב-schema): מחזיר את התיק הקיים
במקום unique-violation.
- precedent_attach — מפתח (case_id, section_id, citation, quote): צירוף חוזר
של אותו ציטוט לאותו סעיף מחזיר את הקיים.
- document_upload — מפתח (case_id, SHA-256 של בייטי הקובץ): העלאה חוזרת של אותו
קובץ מחזירה את המסמך הקיים ו**מדלגת על copy+OCR+embed** (החלק היקר). נוספה
עמודת documents.content_hash (תוספתי, DEFAULT '') + get_document_by_hash.
נבחרה בדיקת-מפתח ברמת-אפליקציה (SELECT-לפני-INSERT) ולא UNIQUE-constraint —
כדי לא לשבור startup אם קיימים נתונים-כפולים legacy. אין מיגרציה הרסנית.
עודכנו docs/spec/X9 (INV-TOOL3 ✅) ו-gap-audit (GAP-52 ✅, פרוסה 2).
py_compile עבר על 4 קבצי הקוד. אימות runtime (restart MCP server) נדחה עד
שהחילוץ הפעיל יסתיים.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
תוספתי בלבד, אפס שבירת-תאימות. שני invariants מחוזה-כלי-ה-MCP (X9):
GAP-44 (INV-TOOL4, סימטריית extract/get): נוסף get_appraiser_facts — ה-get
המקביל ל-extract_appraiser_facts. קורא list_appraiser_facts + detect_appraiser_conflicts
מה-DB ללא חילוץ-LLM יקר ולא-דטרמיניסטי. מחזיר count=0 (לא שגיאה) אם טרם חולץ.
GAP-53 (INV-TOOL5, limit-caps / OWASP API4:2023): נוסף _clamp_limit (תקרה 200,
non-positive→max) על ~13 כלי list/search ב-server.py (case_list, search_*,
precedent_library_list, halachot_pending, missing_precedent_list, list_*_citations…).
list_chair_feedback קיבל param limit חדש (server→workflow→db עם LIMIT) — היה ללא תקרה כלל.
לא הוסף get_appraiser_facts ל-frontmatter של סוכנים (INV-AG3 "לא עודף" — ההוראות
עוד לא מפנות אליו; חיווט = follow-up). נותר ב-FU-14: GAP-45/48/49/50/51/52.
עודכנו docs/spec/X9 (INV-TOOL4/5) ו-gap-audit (סטטוס פרוסה 1).
אומת: py_compile על 4 קבצי הקוד. אימות runtime (restart MCP server) נדחה עד
שהחילוץ הפעיל של היו"ר יסתיים.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
סוגר את לולאת פידבק-יו"ר→ידע-סוכנים. עד כה resolve רק עדכן את ה-DB; עכשיו
לחיצה ב-/feedback מעירה את ה-CEO שמקפל את הלקח לקובץ לפי הקטגוריה.
- paperclip_client.py: wake_ceo_for_feedback_fold() — יוצר issue ב-Paperclip
עם הלקח + rubric ניתוב (style→SKILL.md, wrong_structure→block-schema,
אחר→lessons.md), מעיר CEO. משכפל את דפוס wake_for_precedent_extraction
- db.py: get_chair_feedback(id) — שליפת הערה בודדת עם case_number/appeal_type
- app.py: resolve endpoint מקבל fold (ברירת מחדל true); BackgroundTask
fire-and-forget; guard — רק עם lesson_extracted. מחזיר fold_queued
- legal-ceo.md: dispatch ל-feedback_fold_ + סעיף "קיפול הערת יו"ר" עם rubric
- frontend: useResolveFeedback מקבל fold; /feedback שולח fold=true עם toast;
drafts-panel שולח fold=false (bookkeeping per-case, בלי קיפול כפול)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- בקאנד: GET לפני ה-async task — אם citation כבר קיים כ-external_upload מחזיר 409
- DB: get_external_case_law_by_citation — lookup לפי citation + source_kind
- פרונט: banner אדום עם פרטי הרשומה הקיימת ושני כפתורות:
• "הפעל חילוץ מחדש" — request-halachot ל-ID הקיים וסגירת הטופס
• "מחק את הרשומה" — DELETE עם confirm, ניקוי conflict לאחר מכן
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make the chair's pending-halacha review faster and less exhausting.
Backend:
- New 'deferred' review_status (snooze): stays out of the active library AND
out of the default pending queue, without the finality of 'rejected'.
update_halacha stamps reviewer+reviewed_at on defer; HALACHA_REVIEW_STATUSES
is the single source of valid statuses (PATCH validation now uses it).
- db.update_halachot_batch(ids, status, reviewer) — one atomic UPDATE for a
whole group; invalid status / empty ids are a no-op.
- POST /api/halachot/batch (HalachaBatchReviewRequest) wraps it.
- update_halacha now RETURNs quality_flags too (parity with list_halachot).
Frontend (halacha-review-panel):
- Quality-flag badges (#81: non_decision / truncated_quote / thin_restatement /
quote_unverified) so the chair sees WHY an item was held back.
- Defer action — button + keyboard 'D' — to snooze without rejecting (fixes the
'leave in pending forever' anti-pattern; reject stays the junk verb).
- Per-precedent batch bar: 'אשר הכל' / 'דחה הכל' via useBatchReviewHalachot
(one request, one refetch) with confirm guards.
- Halacha/HalachaPatch types gain quality_flags + 'deferred'.
Verified: mcp-server suite 156 passed; web build green; end-to-end integration
against dev DB (batch approve/reject, defer sets status+timestamp, pending
excludes approved+deferred, deferred queryable, invalid status no-op).
Note: api:types regen deferred until deploy (the batch hook is hand-typed, not
dependent on generated types).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#83 pipeline robustness — the index-numbering correctness guarantee:
- Add CREATE UNIQUE INDEX idx_halachot_unique_index ON halachot(case_law_id,
halacha_index). The extractor assigns the index as MAX+1 under an in-process
store-lock + a cross-process pg advisory lock, so collisions shouldn't occur
in normal operation — but per the research (FireHydrant/OneUptime) the
constraint is the actual correctness guarantee while the lock is the
optimization. A racing/double run now fails LOUDLY (UniqueViolation, chunk
left un-checkpointed → clean resume) instead of silently appending the
duplicates that were the 2026-05/06 over-extraction root cause.
Data prep (run against the live DB before the constraint, backed up to
data/audit/halacha-reindex-backup-*.sql): the 6 precedents that still carried
colliding halacha_index values (9 groups, distinct principles that shared a
number — NOT content dups) were renumbered to unique sequential indices.
Verified: advisory lock holds cross-process and the DB path is direct asyncpg
(no transaction-pooler), so the session lock is safe (83.1); force=True does
delete+checkpoint-clear in one transaction (83.5); constraint rejects a
duplicate-index insert (integration-checked). Full suite 156 passed.
Also commits the TaskMaster tracking for the whole halacha-quality initiative
(#81-#84 + research-backed subtasks, statuses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bake the 2026-06-03 strict-cleanup rubric into the extraction pipeline so the
corpus stays clean at the source instead of accumulating duplicates, obiter
dicta, truncated quotes and thin restatements that clog the review queue.
#81 — quality gate:
- New pure module halacha_quality.py with unit-tested validators:
non-decision/obiter (Wambaugh markers), truncated-quote (mid-word cut),
thin-restatement (rule≈quote), quote-unverified.
- Validators run in halacha_extractor._process; a non-decision is re-typed
obiter; flags persist in new halachot.quality_flags column.
- Auto-approve now requires confidence>=threshold AND no quality flags;
flagged items route to pending_review regardless of confidence.
- Both extraction prompts hardened: reject undecided dicta, exclude
case-specific applications, require abstraction, forbid over-splitting.
#82 — dedup-on-insert (store_halachot_for_chunk):
- Within the same precedent, skip a halacha whose normalized supporting_quote
already exists, or whose rule-embedding has cosine>=HALACHA_DEDUP_COSINE
(0.93) against an already-stored one. Makes re-runs idempotent.
Migration: halachot.quality_flags TEXT[] (additive, idempotent ALTER).
Tests: 19 new unit tests; full suite 156 passed. Validated end-to-end against
dev DB (dedup skips dups, flag blocks auto-approve, re-run inserts 0).
Calibration: flags fire on only ~10% of current survivors (low false-positive).
Spec: docs/halacha-strict-rubric.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Halacha extraction held ALL chunk results in memory and stored once at the very
end — a crash/interrupt mid-run (e.g. the 2026-05-31 freeze) lost everything and
re-paid the full LLM cost on retry.
Now each chunk's halachot are stored AND the chunk is checkpointed
(precedent_chunks.halacha_extracted_at) the moment it finishes:
- V25 schema: precedent_chunks.halacha_extracted_at (per-chunk checkpoint).
- db.store_halachot_for_chunk: atomic per-chunk insert (halacha_index continues
from MAX, caller serializes via an in-process store-lock) + checkpoint mark.
- db.reset_halacha_extraction (force) / mark_all_chunks_extracted (legacy backfill).
- _extract_impl rewritten: resume by default (skip checkpointed chunks; failed
chunks stay pending and are retried; status stays 'processing' until all done);
force=True wipes + redoes all. reextract_halachot passes force=True; the queue
drain (process_pending) resumes by default.
- Legacy guard: a pre-V25 precedent (halachot exist, no checkpoints) is
backfilled and treated as complete — never re-extracted (would duplicate).
Verified on 9002-24 (55 halachot, legacy): resume → legacy-backfill, NO
duplication (stays 55), all chunks checkpointed. Index continuation: store at
55,56 after max 54, no collision. Tracks #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add SCHEMA_V21_SQL (searchable boolean column + index on case_law), wire it
into _run_schema_migrations, and implement _compute_searchable (pure predicate)
+ recompute_searchable (idempotent async backfill/update). All 5 unit tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reported: an agent claimed the case had no documents because document_list
returned empty — but the documents exist. Root cause: get_case_by_number did
an exact `WHERE case_number = $1`, so any formatting variant of the number
silently failed to resolve. Verified on 8137-24 (9 docs): "8137/24",
"ערר 8137-24", leading/trailing space, and "בל\"מ 8126/03/25" all returned
"תיק לא נמצא", which the agent read as "no documents" and went blind.
Add _normalize_case_number (strip leading proceeding-type prefix to the first
digit, trim, unify '/'→'-') and a normalized fallback in the lookup query
(exact match preferred via ORDER BY). One fix covers every case_number-scoped
tool (document_list, extract_references, search_case_documents, get_claims,
drafting, ...). Bogus numbers still correctly resolve to "not found". (#58)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of "agent can't find the Agasi decision in the corpus" (CMPA-55):
the decision was fully ingested, but the retrieval layer failed on the
realistic agent query — searching by case name.
- RC-A (#52): lexical tsvector covered only chunk content + halacha text,
so a bare-name query ("אגסי") matched decisions that *cite* the case, not
the case itself. Add meta_tsv on case_law(case_name, case_number) (SCHEMA
V20) and OR it into the lexical halacha/chunk SQL with a match boost, so a
name/number hit surfaces the case's own rows. Agasi: rank 4 → rank 1.
- RC-B (#53): precedent_library_list hard-defaulted source_kind=external_upload
and never exposed the param, hiding uploaded ערר/בל"מ (internal_committee)
decisions. Thread source_kind through service → tool → MCP tool (supports
'internal_committee' / 'all_committees').
- #54: agent instructions (researcher/analyst/writer) — search-by-name
protocol: add content/case-number, search both corpora, use all_committees
before declaring "not in corpus".
- #55: chunker produced tiny fragment chunks ("דיון", "החלטה") from header
keywords matched mid-sentence. Anchor SECTION_PATTERNS to line start +
merge sub-min sections; exclude <50-char fragments at query time (484
existing fragments hidden; full re-chunk tracked as #57).
Tests: scripts/test_retrieval_by_name.py (name ranks case above citer +
substantive regressions); chunker unit checks (0 tiny chunks). New findings
filed as tasks #56 (halacha source_kind leak) and #57 (re-chunk migration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Six-phase upgrade of /training from a read-only dashboard into a full
Style Studio for managing Daphna's style corpus.
- Upload Sheet on /training: file → proofread preview → commit (no more
CLI-only `upload-training` skill).
- Rich corpus metadata: GET /api/training/corpus returns summary, outcome,
key_principles, page_count, parties (regex), legal_citation, lessons_count.
PATCH endpoint for chair edits. CorpusDetailDrawer with 4 tabs (details
/content/lessons/patterns) replaces the bare table row.
- LLM metadata enrichment: style_metadata_extractor + MCP tools
(style_corpus_enrich, style_corpus_pending_enrichment) fill summary
/outcome/key_principles via claude_session (free, host-side).
- Per-decision lessons: new decision_lessons table + 4 REST endpoints +
LessonsTab in drawer; hermes-curator now auto-posts findings as
decision_lessons(source=curator).
- Curator Portrait tab: prompt rendered with link to Gitea, recent
curator findings, style_analyzer training prompts, propose-change
form that writes proposals to data/curator-proposals/ for manual
chair review (no auto-mutation of the agent file).
- Style chat tab: SSE-streamed conversations with the style agent.
New host-side pm2 service (legal-chat-service, port 8770) wraps
claude CLI with stream-json + --resume continuation; FastAPI proxies
via host.docker.internal. Zero API cost — uses chaim's claude.ai
subscription. chat_conversations + chat_messages persist history.
Architecture: keeps the existing rule that claude_session only runs
on the host (not the container). The new legal-chat-service is the
canonical bridge between the container and the local CLI for the chat
feature; everything else (upload, metadata, lessons) stays within the
container's existing capabilities.
Audit script (scripts/audit_training_corpus.py) included for verifying
which corpus rows still need enrichment.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Until now, "case_number" was the only stored identifier for a precedent.
But a *citation per the Israeli unified citation rules* is a different
beast — it has bold parties, an unbold prefix (court abbrev + panel/
district parenthetical + case number), and an unbold trailing reporter
(נבו / פ"ד...). Without storing it as a first-class field we couldn't
hand the chair a one-click "copy as citation" experience for pasting
into decisions.
Changes:
- Schema V19: case_law.citation_formatted TEXT (Markdown — parties
wrapped in **…** so the copy helper can render <strong> for Word/Docs
paste and keep plain-text fallback meaningful).
- Metadata extractor: composes citation_formatted from the document
text per the unified citation rules, with worked examples for ע"א /
עת"מ / ערר / בל"מ in the prompt. Refuses to store half-formed strings.
- PATCH /api/precedent-library/{id} accepts citation_formatted so the
chair can correct LLM mistakes.
- /precedents/[id]: dedicated "מראה מקום" block with bold rendering,
a copy-to-clipboard button (text/html + text/plain so Word keeps
the bolds), and an inline edit textarea.
- /precedents list rows: link displays the formatted citation when
available, with a small inline copy button — falls back to the bare
case_number for older rows.
Backfill of existing rows happens by re-stamping the extraction queue
once V19 has rolled out and the new field is reachable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## #34 — Daphna's internal citation graph
New schema V16 (V15 was already used by proceeding_type): table
``precedent_internal_citations`` (source→cited, with cited_case_law_id
nullable for citations whose target isn't in the corpus yet) + 3
indexes (source, target, unlinked).
New service ``citation_extractor.py`` with regex patterns for ערר /
בל"מ / עע"מ / בר"מ / עמ"נ / ע"א / בג"ץ / רע"א — accepts both ``\/``
and ``-`` separators, requires actual parenthesized district label
to avoid greedy mid-paragraph captures. Resolves citations against
``case_law.case_number`` substring; default confidence 0.90 linked,
0.75 unlinked. ON CONFLICT DO NOTHING on (source, cited_case_number).
3 new MCP tools: ``extract_internal_citations``,
``list_internal_citations``, ``list_incoming_citations``. Optional
flag ``include_cited_by=True`` on ``search_internal_decisions``
appends cited-by candidates as ``match_type='cited_by'`` stubs.
Bulk-extracted from 40 internal_committee rows authored by דפנה תמיר:
**353 distinct citations, 348 stored, 96 linked / 252 unlinked**.
Top citers: 1079/24 (30), 1024/24 (19), 1009/25 (18). Top unlinked
target: ע"א 3213/97 (cited 5x) — natural #35 candidates.
## #32 — Wide-modal precedent edit
`precedent-edit-sheet.tsx`: ``<Sheet side="left">`` → centered
``<Dialog>`` with ``sm:max-w-4xl`` ``max-h-[90vh]`` ``overflow-y-auto``.
Component API unchanged so existing callers
(`/precedents/[id]/page.tsx`, `library-list-panel.tsx`) work as-is.
RTL preserved. Mobile falls back to near-full-width via shadcn default.
## #13 — 403/17 verification
`case_law e151fc25-...` (אהרון ברק - תכנית רחביה) already in perfect
shape after Stage A work: all metadata fields populated, 351 halachot
with avg_conf=0.864 (well above 0.78 threshold). No re-extraction
needed; closing task as verified.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Same case_number can exist as both a regular appeal (ערר) and an
extension-of-time request (בל"מ), and we were inferring the difference
from appeal_subtype prefixes — fragile, and case-number lookups
weren't disambiguated. Now stored as a first-class field on both
case_law (corpus) and cases (live cases), with partial unique indexes
on (case_number, proceeding_type).
- SCHEMA_V15: column + CHECK constraints + backfill from
appeal_subtype LIKE 'extension_request_%' + partial unique indexes
replace the old global UNIQUE(case_number).
- derive_proceeding_type() centralizes the inference rule
(extension_request_* → בל"מ; subject regex fallback; default ערר).
- Metadata extractor prompt asks Claude to populate the new field
explicitly; apply_to_record writes it for internal_committee rows.
- internal_decision_upload, case_create, case_update accept an
optional proceeding_type; FastAPI request models expose it.
- Wizard + edit dialog get a sided Select; case header renders the
resolved label (ערר / בל"מ).
- Uploaded the 2 staged בל"מ decisions on betterment levy:
8126/24 (סופר נוח, 13 chunks), 8047/23 (הרנון, 48 chunks).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four parallel sub-agents closed the remaining critical gaps from the
26/05 Stage A/B sprint. Each block independently tested; aggregated here.
## #30/#31 finalizers (sub-agent A)
* Auto-derive practice_area in case_create from case_number prefix
(1xxx→rishuy_uvniya, 8xxx→betterment_levy, 9xxx→compensation_197);
default for CaseCreateRequest is now "" (the DB constraint catches
any stray "appeals_committee").
* practice_area.py: derive_subtype now handles axis-B domain values
(rishuy_uvniya/betterment_levy/compensation_197) without parsing the
case number; new helper derive_domain_practice_area().
* Halacha re-extraction verified unnecessary — all 6 reclassified
records already had is_binding=false and approved halachot.
* Regression tests: 6 cases in tests/test_corpus_constraints.py
covering practice_area enum, internal-committee chair/district,
external-upload arar prefix, MCP guard.
* UI: district input → Select dropdown (7 districts) in
precedent-edit-sheet.tsx, preserving legacy free-text values.
## #37 בל"מ subtypes (sub-agent B)
* 3 new appeal_subtypes: extension_request_{building_permit,
betterment_levy,compensation}. APPEALS_COMMITTEE_SUBTYPES extended,
SUBTYPES_BY_AREA mappings added.
* New helpers: is_blam_subject(), is_blam_subtype(),
derive_subtype_with_blam(case_number, subject, practice_area).
case_create now uses it to auto-detect "בקשה להארכת מועד" subjects.
* 3 methodology templates under docs/methodology/extension-request-*.md.
* paperclip_client.py mapping updated for the 3 new subtypes
(extension_request_building_permit→CMP, the other two→CMPA).
* Frontend: bilingual "בל"מ" badge + filter dropdown on cases list +
detail header; appeal-type-bars collapseBlam() merges בל"מ into its
parent domain for aggregate bars.
* Wizard auto-detects בל"מ from subject during case creation.
* 3 Berlinger cases (1017/1018/1019-03-26) migrated to
appeal_subtype=extension_request_building_permit via psql.
## #35 missing_precedents feature (sub-agent C)
* Schema V13: missing_precedents table (citation, case_id, party,
legal_topic, status, linked_case_law_id, claim_quote, ...) +
FK constraints + 3 indexes. Applied via psql + idempotent migration.
* 6 db.py service functions, 3 MCP tools, 6 FastAPI endpoints
(POST/GET/PATCH/DELETE/upload — upload routes by citation prefix
to ingest_internal_decision or ingest_precedent).
* Next.js page /missing-precedents with 5 status tabs + filters +
sidebar badge counter + detail drawer with metadata edit + smart
upload form that switches fields per committee/court.
* Bootstrap: 7 rows imported from the JSON file
(3 citations × cases, all status=closed with linked_case_law_id).
* legal-researcher.md: new §2ב.5 with missing_precedent_create
usage + dedup semantics + tool grant.
## #36 legal_arguments aggregation (sub-agent D)
* Schema V14: legal_arguments + legal_argument_propositions M:M.
Applied via psql.
* New service argument_aggregator.py with two functions —
aggregate_claims_to_arguments() (Claude CLI / claude_session) and
get_legal_arguments(). Graceful llm_unavailable handling when CLI
is missing (containers).
* 2 MCP tools + 2 API endpoints (POST .../aggregate-arguments as
BackgroundTask, GET .../legal-arguments).
* Frontend: shadcn Accordion + new legal-arguments-panel.tsx with
hierarchical (party → priority badge → arguments) display, "טיעונים"
tab on the case page, "חשב/חשב מחדש" buttons.
* scripts/backfill_legal_arguments.py + SCRIPTS.md entry — dry-run
found 8 candidate cases including 1017/1018/1019.
## Open follow-ups (intentionally deferred)
* npm run api:types in web-ui (CLAUDE.md flow) — recommended before
the next UI commit; not required for backend deployment.
* Run backfill_legal_arguments.py --apply once the container picks up
the new aggregator service.
* webhook on missing-precedents upload-close to Paperclip (optional).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five enhancements to the precedent retrieval stack:
* **#44 HNSW indexes** for precedent_chunks + halachot (replacing IVFFlat
lists=50). Build time ~3s combined. Better recall@10 with pgvector 0.8.2.
* **#45 Halacha sweep** — 96 pending halachot at conf>=0.78 promoted to
approved (1141 → 1237). Cluster at conf=0.78 spot-checked OK. Applied
via psql only — env HALACHA_AUTO_APPROVE_THRESHOLD unchanged (0.80).
* **#43 MMR diversity** — search_precedent_library_hybrid now caps at
``max_per_case_law=2`` (default). Prevents one precedent dominating
top-10 when many of its chunks/halachot rank high. New helper
``_diversify_by_case_law`` in hybrid_search.py.
* **#46 Dynamic halacha boost** — replaces the static ``score+=0.05``
with ``score+=confidence*0.06``. Calibrated so avg-confidence (~0.85)
stays at +0.05; high-conf halachot get a slight extra lift, low-conf
ones get less. Behaviour preserved at the mean.
* **#41 BM25/tsvector hybrid + RRF**. Schema V12 adds STORED tsvector
columns ``precedent_chunks.content_tsv`` and ``halachot.rule_tsv``
(using simple config — Postgres has no Hebrew stemmer) + GIN indexes.
New ``db.search_precedent_library_lexical`` mirrors the semantic
function with ts_rank_cd over plainto_tsquery. ``hybrid_search``
runs sem+lex in parallel and fuses via RRF before rerank. Toggle:
env ``BM25_HYBRID_ENABLED`` (default true), graceful fallback to
semantic-only on lexical failure.
#40 (VOYAGE_RERANK_ENABLED) was already true in Coolify env; no change.
#42 (Claude Haiku query expansion) deferred — latency + cost concerns
warrant a separate plan; the bm25 lexical leg already recovers most of
the exact-string recall #42 was meant to address.
Closes TaskMaster #41, #43-#46.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fields existed in DB and Precedent type but were missing from:
- PrecedentUpdateRequest (backend model)
- update_case_law allowed set (db layer)
- PrecedentPatch (frontend type)
- precedent-edit-sheet form state, inputs, and patch payload
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- DB: add 'all_committees' virtual source_kind covering internal_committee
+ external_upload appeals_committee rows in one query
- DB: stats now count all case_law rows (not just external_upload),
fixing the precedents_total that excluded 44 internal-committee records
- UI: courts table filters to source_type=court_ruling only;
committees table uses the new all_committees query
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add ability to mark case_law records as related (e.g. same appeal
through ועדת ערר → מנהלי → עליון):
- DB: case_law_relations join table (bidirectional, V11 migration)
- DB CRUD: add/remove/get_case_law_relations
- Service: get_precedent() now returns related_cases[]
- MCP: precedent_link_cases + precedent_unlink_cases tools
- REST: POST/DELETE /api/precedent-library/{id}/relations
- UI: RelatedCasesSection on detail page with search dialog and unlink
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Earlier commit afcc481 opened request_metadata_extraction and
request_halacha_extraction to all source kinds — but
list_pending_extraction_requests still hard-filtered to external_upload.
Result: stamping a queue request on an internal_committee row succeeded
silently, but the worker (and the queue badge) never saw it. Even with
the auto-wakeup added in c7132ba the CEO would wake, find 0 pending
items, and exit.
Drop the legacy filter so the queue listing matches the writer side.
Coolify deploy required for the FastAPI container to pick this up.
