asyncpg עם pgvector register_vector מקבל את ה-embedding כ-list[float] ישירות;
str() גרם ל-DataError. תוקן בהתאם לדפוס store_*_image_embeddings.
Backfill הורץ בהצלחה: 2670 דוגמאות מ-83 החלטות.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
סוגר את לולאת-הלמידה (INV-LRN4): כל החלטה נסגרת מול הסופי, וכל סופי
מנותח מול הטיוטה. מזין את הטבלאות ש-T15 כבר קורא מהן.
T5 — פנקס-התאמה:
- SCHEMA_V26: טבלת draft_final_pairs (snapshot draft + final + diff + analysis + status).
- db: create/update/list_draft_final_pairs.
- mark-final (app.py): תופס snapshot של הטיוטה (decision_blocks) ברגע החתימה,
לפני שאפשר לדרוס אותו, ופותח שורת-פנקס (status=final_received).
T4 — דיסטילציה אוטומטית:
- learning_loop.process_final_version: משתמש ב-snapshot (לא בבלוקים שאולי השתנו),
מסווג style_method↔substance, שומר הצעה ב-pair (status=analyzed).
**הוסר ה-auto-upsert של style_patterns** — ביטל את ה-bug שדרס את שער-היו"ר
וזיהם סגנון במהות (INV-LRN1 + INV-LRN5).
- LESSONS_PROMPT: הפרדת style_method↔substance מפורשת + לקח מופשט בלבד.
- curator wake + hermes-curator.md: מריץ ingest_final_version ראשון; מציע רק
style_method שלא תועד; substance→מסלול precedent.
INV-LRN1 (שער-יו"ר, אין auto-commit) · INV-LRN4 (ניגוד-אמת) · INV-LRN5 (טוהר).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
עונה ל"להתחשב במה שכבר למדנו": הכותב התעלם מעריכות היו"ר ב-/methodology
(נשמרו ב-appeal_type_rules אך block_writer קרא רק קבועי lessons.py) ומ-
decision_lessons של /training. עכשיו הכל מגיע לכתיבה.
- db.get_methodology_overrides(category) — overrides של היו"ר (יחסי-זהב,
כללי-דיון, צ׳קליסטים) מ-appeal_type_rules (כמו merge של ה-API).
- db.get_recent_decision_lessons(limit, practice_area) — לקחי /training.
- _build_style_context(practice_area): מוסיף סעיף "⭐ למידה מצטברת — גובר
על ברירת-מחדל" עם שניהם, אחרי voice-fingerprint (T0). שני ה-callers מעבירים
practice_area. עובד יחד עם הלולאה (T4/T5) שתזין לאותן טבלאות.
תיקון-מספור (חלק מ-T9, דחוף כי T0 הזריק את הטעות): voice-fingerprint §3.1
תוקן — ההחלטה ממוספרת תמיד (מספור-אוטומטי ב-Word); "ללא מספור" היה
ארטיפקט-חילוץ. האנטי-דפוס האמיתי: רשימת-מיני בתוך פסקה + מספרים ידניים.
INV-LRN4 (הזרמת למידה) · INV-LRN5 (טוהר). G11.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
שתי בעיות UX בדף /precedents:
1. חילוץ מטא-דאטה לא נתן שום אינדיקציה שהוא רץ. בניגוד לחילוץ טקסט/הלכות
(extraction_status / halacha_extraction_status) למטא-דאטה היתה רק חותמת-זמן
metadata_extraction_requested_at — אין מצב "processing", לכן StatusPill לא
הציג כלום. נוספה עמודת metadata_extraction_status ('pending'|'processing'|
'completed'|'failed') במתכונת העמודות הקיימות, וה-worker
(process_pending_extractions + reextract_metadata) מעדכן אותה: processing
בתחילת פריט, completed בסיום (מנקה גם את החותמת), pending בכשל (לריטריי).
ה-UI מציג תג "מחלץ מטא-דאטה" + באנר מונה-אצווה עם אחוז התקדמות (high-water-mark
של עומק-התור) שמתעדכן אוטומטית דרך ה-polling הקיים (5ש').
2. שתי טבלאות מוערמות (בתי משפט / ועדות ערר) חייבו גלילה ארוכה. הוחלפו במתג-
מקטעים — טבלה אחת בכל פעם, עם שמירה על העמודות הייעודיות לכל סוג.
Invariants: G2 (מרחיב מנגנון-סטטוס קיים, לא מסלול מקביל), INV-TOOL4/GAP-45
(המשך חשיפת תור-החילוץ הסמוי). אין נגיעה בתוכן משפטי (G11).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INV-TOOL3 (idempotency על מפתח דטרמיניסטי). כל שלושת הכלים מחזירים את הרשומה
הקיימת במקום ליצור כפילות:
- case_create — מפתח case_number (כבר UNIQUE ב-schema): מחזיר את התיק הקיים
במקום unique-violation.
- precedent_attach — מפתח (case_id, section_id, citation, quote): צירוף חוזר
של אותו ציטוט לאותו סעיף מחזיר את הקיים.
- document_upload — מפתח (case_id, SHA-256 של בייטי הקובץ): העלאה חוזרת של אותו
קובץ מחזירה את המסמך הקיים ו**מדלגת על copy+OCR+embed** (החלק היקר). נוספה
עמודת documents.content_hash (תוספתי, DEFAULT '') + get_document_by_hash.
נבחרה בדיקת-מפתח ברמת-אפליקציה (SELECT-לפני-INSERT) ולא UNIQUE-constraint —
כדי לא לשבור startup אם קיימים נתונים-כפולים legacy. אין מיגרציה הרסנית.
עודכנו docs/spec/X9 (INV-TOOL3 ✅) ו-gap-audit (GAP-52 ✅, פרוסה 2).
py_compile עבר על 4 קבצי הקוד. אימות runtime (restart MCP server) נדחה עד
שהחילוץ הפעיל יסתיים.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
תוספתי בלבד, אפס שבירת-תאימות. שני invariants מחוזה-כלי-ה-MCP (X9):
GAP-44 (INV-TOOL4, סימטריית extract/get): נוסף get_appraiser_facts — ה-get
המקביל ל-extract_appraiser_facts. קורא list_appraiser_facts + detect_appraiser_conflicts
מה-DB ללא חילוץ-LLM יקר ולא-דטרמיניסטי. מחזיר count=0 (לא שגיאה) אם טרם חולץ.
GAP-53 (INV-TOOL5, limit-caps / OWASP API4:2023): נוסף _clamp_limit (תקרה 200,
non-positive→max) על ~13 כלי list/search ב-server.py (case_list, search_*,
precedent_library_list, halachot_pending, missing_precedent_list, list_*_citations…).
list_chair_feedback קיבל param limit חדש (server→workflow→db עם LIMIT) — היה ללא תקרה כלל.
לא הוסף get_appraiser_facts ל-frontmatter של סוכנים (INV-AG3 "לא עודף" — ההוראות
עוד לא מפנות אליו; חיווט = follow-up). נותר ב-FU-14: GAP-45/48/49/50/51/52.
עודכנו docs/spec/X9 (INV-TOOL4/5) ו-gap-audit (סטטוס פרוסה 1).
אומת: py_compile על 4 קבצי הקוד. אימות runtime (restart MCP server) נדחה עד
שהחילוץ הפעיל של היו"ר יסתיים.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
סוגר את לולאת פידבק-יו"ר→ידע-סוכנים. עד כה resolve רק עדכן את ה-DB; עכשיו
לחיצה ב-/feedback מעירה את ה-CEO שמקפל את הלקח לקובץ לפי הקטגוריה.
- paperclip_client.py: wake_ceo_for_feedback_fold() — יוצר issue ב-Paperclip
עם הלקח + rubric ניתוב (style→SKILL.md, wrong_structure→block-schema,
אחר→lessons.md), מעיר CEO. משכפל את דפוס wake_for_precedent_extraction
- db.py: get_chair_feedback(id) — שליפת הערה בודדת עם case_number/appeal_type
- app.py: resolve endpoint מקבל fold (ברירת מחדל true); BackgroundTask
fire-and-forget; guard — רק עם lesson_extracted. מחזיר fold_queued
- legal-ceo.md: dispatch ל-feedback_fold_ + סעיף "קיפול הערת יו"ר" עם rubric
- frontend: useResolveFeedback מקבל fold; /feedback שולח fold=true עם toast;
drafts-panel שולח fold=false (bookkeeping per-case, בלי קיפול כפול)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- בקאנד: GET לפני ה-async task — אם citation כבר קיים כ-external_upload מחזיר 409
- DB: get_external_case_law_by_citation — lookup לפי citation + source_kind
- פרונט: banner אדום עם פרטי הרשומה הקיימת ושני כפתורות:
• "הפעל חילוץ מחדש" — request-halachot ל-ID הקיים וסגירת הטופס
• "מחק את הרשומה" — DELETE עם confirm, ניקוי conflict לאחר מכן
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make the chair's pending-halacha review faster and less exhausting.
Backend:
- New 'deferred' review_status (snooze): stays out of the active library AND
out of the default pending queue, without the finality of 'rejected'.
update_halacha stamps reviewer+reviewed_at on defer; HALACHA_REVIEW_STATUSES
is the single source of valid statuses (PATCH validation now uses it).
- db.update_halachot_batch(ids, status, reviewer) — one atomic UPDATE for a
whole group; invalid status / empty ids are a no-op.
- POST /api/halachot/batch (HalachaBatchReviewRequest) wraps it.
- update_halacha now RETURNs quality_flags too (parity with list_halachot).
Frontend (halacha-review-panel):
- Quality-flag badges (#81: non_decision / truncated_quote / thin_restatement /
quote_unverified) so the chair sees WHY an item was held back.
- Defer action — button + keyboard 'D' — to snooze without rejecting (fixes the
'leave in pending forever' anti-pattern; reject stays the junk verb).
- Per-precedent batch bar: 'אשר הכל' / 'דחה הכל' via useBatchReviewHalachot
(one request, one refetch) with confirm guards.
- Halacha/HalachaPatch types gain quality_flags + 'deferred'.
Verified: mcp-server suite 156 passed; web build green; end-to-end integration
against dev DB (batch approve/reject, defer sets status+timestamp, pending
excludes approved+deferred, deferred queryable, invalid status no-op).
Note: api:types regen deferred until deploy (the batch hook is hand-typed, not
dependent on generated types).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#83 pipeline robustness — the index-numbering correctness guarantee:
- Add CREATE UNIQUE INDEX idx_halachot_unique_index ON halachot(case_law_id,
halacha_index). The extractor assigns the index as MAX+1 under an in-process
store-lock + a cross-process pg advisory lock, so collisions shouldn't occur
in normal operation — but per the research (FireHydrant/OneUptime) the
constraint is the actual correctness guarantee while the lock is the
optimization. A racing/double run now fails LOUDLY (UniqueViolation, chunk
left un-checkpointed → clean resume) instead of silently appending the
duplicates that were the 2026-05/06 over-extraction root cause.
Data prep (run against the live DB before the constraint, backed up to
data/audit/halacha-reindex-backup-*.sql): the 6 precedents that still carried
colliding halacha_index values (9 groups, distinct principles that shared a
number — NOT content dups) were renumbered to unique sequential indices.
Verified: advisory lock holds cross-process and the DB path is direct asyncpg
(no transaction-pooler), so the session lock is safe (83.1); force=True does
delete+checkpoint-clear in one transaction (83.5); constraint rejects a
duplicate-index insert (integration-checked). Full suite 156 passed.
Also commits the TaskMaster tracking for the whole halacha-quality initiative
(#81-#84 + research-backed subtasks, statuses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bake the 2026-06-03 strict-cleanup rubric into the extraction pipeline so the
corpus stays clean at the source instead of accumulating duplicates, obiter
dicta, truncated quotes and thin restatements that clog the review queue.
#81 — quality gate:
- New pure module halacha_quality.py with unit-tested validators:
non-decision/obiter (Wambaugh markers), truncated-quote (mid-word cut),
thin-restatement (rule≈quote), quote-unverified.
- Validators run in halacha_extractor._process; a non-decision is re-typed
obiter; flags persist in new halachot.quality_flags column.
- Auto-approve now requires confidence>=threshold AND no quality flags;
flagged items route to pending_review regardless of confidence.
- Both extraction prompts hardened: reject undecided dicta, exclude
case-specific applications, require abstraction, forbid over-splitting.
#82 — dedup-on-insert (store_halachot_for_chunk):
- Within the same precedent, skip a halacha whose normalized supporting_quote
already exists, or whose rule-embedding has cosine>=HALACHA_DEDUP_COSINE
(0.93) against an already-stored one. Makes re-runs idempotent.
Migration: halachot.quality_flags TEXT[] (additive, idempotent ALTER).
Tests: 19 new unit tests; full suite 156 passed. Validated end-to-end against
dev DB (dedup skips dups, flag blocks auto-approve, re-run inserts 0).
Calibration: flags fire on only ~10% of current survivors (low false-positive).
Spec: docs/halacha-strict-rubric.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Halacha extraction held ALL chunk results in memory and stored once at the very
end — a crash/interrupt mid-run (e.g. the 2026-05-31 freeze) lost everything and
re-paid the full LLM cost on retry.
Now each chunk's halachot are stored AND the chunk is checkpointed
(precedent_chunks.halacha_extracted_at) the moment it finishes:
- V25 schema: precedent_chunks.halacha_extracted_at (per-chunk checkpoint).
- db.store_halachot_for_chunk: atomic per-chunk insert (halacha_index continues
from MAX, caller serializes via an in-process store-lock) + checkpoint mark.
- db.reset_halacha_extraction (force) / mark_all_chunks_extracted (legacy backfill).
- _extract_impl rewritten: resume by default (skip checkpointed chunks; failed
chunks stay pending and are retried; status stays 'processing' until all done);
force=True wipes + redoes all. reextract_halachot passes force=True; the queue
drain (process_pending) resumes by default.
- Legacy guard: a pre-V25 precedent (halachot exist, no checkpoints) is
backfilled and treated as complete — never re-extracted (would duplicate).
Verified on 9002-24 (55 halachot, legacy): resume → legacy-backfill, NO
duplication (stays 55), all chunks checkpointed. Index continuation: store at
55,56 after max 54, no collision. Tracks #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add SCHEMA_V21_SQL (searchable boolean column + index on case_law), wire it
into _run_schema_migrations, and implement _compute_searchable (pure predicate)
+ recompute_searchable (idempotent async backfill/update). All 5 unit tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reported: an agent claimed the case had no documents because document_list
returned empty — but the documents exist. Root cause: get_case_by_number did
an exact `WHERE case_number = $1`, so any formatting variant of the number
silently failed to resolve. Verified on 8137-24 (9 docs): "8137/24",
"ערר 8137-24", leading/trailing space, and "בל\"מ 8126/03/25" all returned
"תיק לא נמצא", which the agent read as "no documents" and went blind.
Add _normalize_case_number (strip leading proceeding-type prefix to the first
digit, trim, unify '/'→'-') and a normalized fallback in the lookup query
(exact match preferred via ORDER BY). One fix covers every case_number-scoped
tool (document_list, extract_references, search_case_documents, get_claims,
drafting, ...). Bogus numbers still correctly resolve to "not found". (#58)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of "agent can't find the Agasi decision in the corpus" (CMPA-55):
the decision was fully ingested, but the retrieval layer failed on the
realistic agent query — searching by case name.
- RC-A (#52): lexical tsvector covered only chunk content + halacha text,
so a bare-name query ("אגסי") matched decisions that *cite* the case, not
the case itself. Add meta_tsv on case_law(case_name, case_number) (SCHEMA
V20) and OR it into the lexical halacha/chunk SQL with a match boost, so a
name/number hit surfaces the case's own rows. Agasi: rank 4 → rank 1.
- RC-B (#53): precedent_library_list hard-defaulted source_kind=external_upload
and never exposed the param, hiding uploaded ערר/בל"מ (internal_committee)
decisions. Thread source_kind through service → tool → MCP tool (supports
'internal_committee' / 'all_committees').
- #54: agent instructions (researcher/analyst/writer) — search-by-name
protocol: add content/case-number, search both corpora, use all_committees
before declaring "not in corpus".
- #55: chunker produced tiny fragment chunks ("דיון", "החלטה") from header
keywords matched mid-sentence. Anchor SECTION_PATTERNS to line start +
merge sub-min sections; exclude <50-char fragments at query time (484
existing fragments hidden; full re-chunk tracked as #57).
Tests: scripts/test_retrieval_by_name.py (name ranks case above citer +
substantive regressions); chunker unit checks (0 tiny chunks). New findings
filed as tasks #56 (halacha source_kind leak) and #57 (re-chunk migration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Six-phase upgrade of /training from a read-only dashboard into a full
Style Studio for managing Daphna's style corpus.
- Upload Sheet on /training: file → proofread preview → commit (no more
CLI-only `upload-training` skill).
- Rich corpus metadata: GET /api/training/corpus returns summary, outcome,
key_principles, page_count, parties (regex), legal_citation, lessons_count.
PATCH endpoint for chair edits. CorpusDetailDrawer with 4 tabs (details
/content/lessons/patterns) replaces the bare table row.
- LLM metadata enrichment: style_metadata_extractor + MCP tools
(style_corpus_enrich, style_corpus_pending_enrichment) fill summary
/outcome/key_principles via claude_session (free, host-side).
- Per-decision lessons: new decision_lessons table + 4 REST endpoints +
LessonsTab in drawer; hermes-curator now auto-posts findings as
decision_lessons(source=curator).
- Curator Portrait tab: prompt rendered with link to Gitea, recent
curator findings, style_analyzer training prompts, propose-change
form that writes proposals to data/curator-proposals/ for manual
chair review (no auto-mutation of the agent file).
- Style chat tab: SSE-streamed conversations with the style agent.
New host-side pm2 service (legal-chat-service, port 8770) wraps
claude CLI with stream-json + --resume continuation; FastAPI proxies
via host.docker.internal. Zero API cost — uses chaim's claude.ai
subscription. chat_conversations + chat_messages persist history.
Architecture: keeps the existing rule that claude_session only runs
on the host (not the container). The new legal-chat-service is the
canonical bridge between the container and the local CLI for the chat
feature; everything else (upload, metadata, lessons) stays within the
container's existing capabilities.
Audit script (scripts/audit_training_corpus.py) included for verifying
which corpus rows still need enrichment.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Until now, "case_number" was the only stored identifier for a precedent.
But a *citation per the Israeli unified citation rules* is a different
beast — it has bold parties, an unbold prefix (court abbrev + panel/
district parenthetical + case number), and an unbold trailing reporter
(נבו / פ"ד...). Without storing it as a first-class field we couldn't
hand the chair a one-click "copy as citation" experience for pasting
into decisions.
Changes:
- Schema V19: case_law.citation_formatted TEXT (Markdown — parties
wrapped in **…** so the copy helper can render <strong> for Word/Docs
paste and keep plain-text fallback meaningful).
- Metadata extractor: composes citation_formatted from the document
text per the unified citation rules, with worked examples for ע"א /
עת"מ / ערר / בל"מ in the prompt. Refuses to store half-formed strings.
- PATCH /api/precedent-library/{id} accepts citation_formatted so the
chair can correct LLM mistakes.
- /precedents/[id]: dedicated "מראה מקום" block with bold rendering,
a copy-to-clipboard button (text/html + text/plain so Word keeps
the bolds), and an inline edit textarea.
- /precedents list rows: link displays the formatted citation when
available, with a small inline copy button — falls back to the bare
case_number for older rows.
Backfill of existing rows happens by re-stamping the extraction queue
once V19 has rolled out and the new field is reachable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## #34 — Daphna's internal citation graph
New schema V16 (V15 was already used by proceeding_type): table
``precedent_internal_citations`` (source→cited, with cited_case_law_id
nullable for citations whose target isn't in the corpus yet) + 3
indexes (source, target, unlinked).
New service ``citation_extractor.py`` with regex patterns for ערר /
בל"מ / עע"מ / בר"מ / עמ"נ / ע"א / בג"ץ / רע"א — accepts both ``\/``
and ``-`` separators, requires actual parenthesized district label
to avoid greedy mid-paragraph captures. Resolves citations against
``case_law.case_number`` substring; default confidence 0.90 linked,
0.75 unlinked. ON CONFLICT DO NOTHING on (source, cited_case_number).
3 new MCP tools: ``extract_internal_citations``,
``list_internal_citations``, ``list_incoming_citations``. Optional
flag ``include_cited_by=True`` on ``search_internal_decisions``
appends cited-by candidates as ``match_type='cited_by'`` stubs.
Bulk-extracted from 40 internal_committee rows authored by דפנה תמיר:
**353 distinct citations, 348 stored, 96 linked / 252 unlinked**.
Top citers: 1079/24 (30), 1024/24 (19), 1009/25 (18). Top unlinked
target: ע"א 3213/97 (cited 5x) — natural #35 candidates.
## #32 — Wide-modal precedent edit
`precedent-edit-sheet.tsx`: ``<Sheet side="left">`` → centered
``<Dialog>`` with ``sm:max-w-4xl`` ``max-h-[90vh]`` ``overflow-y-auto``.
Component API unchanged so existing callers
(`/precedents/[id]/page.tsx`, `library-list-panel.tsx`) work as-is.
RTL preserved. Mobile falls back to near-full-width via shadcn default.
## #13 — 403/17 verification
`case_law e151fc25-...` (אהרון ברק - תכנית רחביה) already in perfect
shape after Stage A work: all metadata fields populated, 351 halachot
with avg_conf=0.864 (well above 0.78 threshold). No re-extraction
needed; closing task as verified.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Same case_number can exist as both a regular appeal (ערר) and an
extension-of-time request (בל"מ), and we were inferring the difference
from appeal_subtype prefixes — fragile, and case-number lookups
weren't disambiguated. Now stored as a first-class field on both
case_law (corpus) and cases (live cases), with partial unique indexes
on (case_number, proceeding_type).
- SCHEMA_V15: column + CHECK constraints + backfill from
appeal_subtype LIKE 'extension_request_%' + partial unique indexes
replace the old global UNIQUE(case_number).
- derive_proceeding_type() centralizes the inference rule
(extension_request_* → בל"מ; subject regex fallback; default ערר).
- Metadata extractor prompt asks Claude to populate the new field
explicitly; apply_to_record writes it for internal_committee rows.
- internal_decision_upload, case_create, case_update accept an
optional proceeding_type; FastAPI request models expose it.
- Wizard + edit dialog get a sided Select; case header renders the
resolved label (ערר / בל"מ).
- Uploaded the 2 staged בל"מ decisions on betterment levy:
8126/24 (סופר נוח, 13 chunks), 8047/23 (הרנון, 48 chunks).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four parallel sub-agents closed the remaining critical gaps from the
26/05 Stage A/B sprint. Each block independently tested; aggregated here.
## #30/#31 finalizers (sub-agent A)
* Auto-derive practice_area in case_create from case_number prefix
(1xxx→rishuy_uvniya, 8xxx→betterment_levy, 9xxx→compensation_197);
default for CaseCreateRequest is now "" (the DB constraint catches
any stray "appeals_committee").
* practice_area.py: derive_subtype now handles axis-B domain values
(rishuy_uvniya/betterment_levy/compensation_197) without parsing the
case number; new helper derive_domain_practice_area().
* Halacha re-extraction verified unnecessary — all 6 reclassified
records already had is_binding=false and approved halachot.
* Regression tests: 6 cases in tests/test_corpus_constraints.py
covering practice_area enum, internal-committee chair/district,
external-upload arar prefix, MCP guard.
* UI: district input → Select dropdown (7 districts) in
precedent-edit-sheet.tsx, preserving legacy free-text values.
## #37 בל"מ subtypes (sub-agent B)
* 3 new appeal_subtypes: extension_request_{building_permit,
betterment_levy,compensation}. APPEALS_COMMITTEE_SUBTYPES extended,
SUBTYPES_BY_AREA mappings added.
* New helpers: is_blam_subject(), is_blam_subtype(),
derive_subtype_with_blam(case_number, subject, practice_area).
case_create now uses it to auto-detect "בקשה להארכת מועד" subjects.
* 3 methodology templates under docs/methodology/extension-request-*.md.
* paperclip_client.py mapping updated for the 3 new subtypes
(extension_request_building_permit→CMP, the other two→CMPA).
* Frontend: bilingual "בל"מ" badge + filter dropdown on cases list +
detail header; appeal-type-bars collapseBlam() merges בל"מ into its
parent domain for aggregate bars.
* Wizard auto-detects בל"מ from subject during case creation.
* 3 Berlinger cases (1017/1018/1019-03-26) migrated to
appeal_subtype=extension_request_building_permit via psql.
## #35 missing_precedents feature (sub-agent C)
* Schema V13: missing_precedents table (citation, case_id, party,
legal_topic, status, linked_case_law_id, claim_quote, ...) +
FK constraints + 3 indexes. Applied via psql + idempotent migration.
* 6 db.py service functions, 3 MCP tools, 6 FastAPI endpoints
(POST/GET/PATCH/DELETE/upload — upload routes by citation prefix
to ingest_internal_decision or ingest_precedent).
* Next.js page /missing-precedents with 5 status tabs + filters +
sidebar badge counter + detail drawer with metadata edit + smart
upload form that switches fields per committee/court.
* Bootstrap: 7 rows imported from the JSON file
(3 citations × cases, all status=closed with linked_case_law_id).
* legal-researcher.md: new §2ב.5 with missing_precedent_create
usage + dedup semantics + tool grant.
## #36 legal_arguments aggregation (sub-agent D)
* Schema V14: legal_arguments + legal_argument_propositions M:M.
Applied via psql.
* New service argument_aggregator.py with two functions —
aggregate_claims_to_arguments() (Claude CLI / claude_session) and
get_legal_arguments(). Graceful llm_unavailable handling when CLI
is missing (containers).
* 2 MCP tools + 2 API endpoints (POST .../aggregate-arguments as
BackgroundTask, GET .../legal-arguments).
* Frontend: shadcn Accordion + new legal-arguments-panel.tsx with
hierarchical (party → priority badge → arguments) display, "טיעונים"
tab on the case page, "חשב/חשב מחדש" buttons.
* scripts/backfill_legal_arguments.py + SCRIPTS.md entry — dry-run
found 8 candidate cases including 1017/1018/1019.
## Open follow-ups (intentionally deferred)
* npm run api:types in web-ui (CLAUDE.md flow) — recommended before
the next UI commit; not required for backend deployment.
* Run backfill_legal_arguments.py --apply once the container picks up
the new aggregator service.
* webhook on missing-precedents upload-close to Paperclip (optional).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five enhancements to the precedent retrieval stack:
* **#44 HNSW indexes** for precedent_chunks + halachot (replacing IVFFlat
lists=50). Build time ~3s combined. Better recall@10 with pgvector 0.8.2.
* **#45 Halacha sweep** — 96 pending halachot at conf>=0.78 promoted to
approved (1141 → 1237). Cluster at conf=0.78 spot-checked OK. Applied
via psql only — env HALACHA_AUTO_APPROVE_THRESHOLD unchanged (0.80).
* **#43 MMR diversity** — search_precedent_library_hybrid now caps at
``max_per_case_law=2`` (default). Prevents one precedent dominating
top-10 when many of its chunks/halachot rank high. New helper
``_diversify_by_case_law`` in hybrid_search.py.
* **#46 Dynamic halacha boost** — replaces the static ``score+=0.05``
with ``score+=confidence*0.06``. Calibrated so avg-confidence (~0.85)
stays at +0.05; high-conf halachot get a slight extra lift, low-conf
ones get less. Behaviour preserved at the mean.
* **#41 BM25/tsvector hybrid + RRF**. Schema V12 adds STORED tsvector
columns ``precedent_chunks.content_tsv`` and ``halachot.rule_tsv``
(using simple config — Postgres has no Hebrew stemmer) + GIN indexes.
New ``db.search_precedent_library_lexical`` mirrors the semantic
function with ts_rank_cd over plainto_tsquery. ``hybrid_search``
runs sem+lex in parallel and fuses via RRF before rerank. Toggle:
env ``BM25_HYBRID_ENABLED`` (default true), graceful fallback to
semantic-only on lexical failure.
#40 (VOYAGE_RERANK_ENABLED) was already true in Coolify env; no change.
#42 (Claude Haiku query expansion) deferred — latency + cost concerns
warrant a separate plan; the bm25 lexical leg already recovers most of
the exact-string recall #42 was meant to address.
Closes TaskMaster #41, #43-#46.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fields existed in DB and Precedent type but were missing from:
- PrecedentUpdateRequest (backend model)
- update_case_law allowed set (db layer)
- PrecedentPatch (frontend type)
- precedent-edit-sheet form state, inputs, and patch payload
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- DB: add 'all_committees' virtual source_kind covering internal_committee
+ external_upload appeals_committee rows in one query
- DB: stats now count all case_law rows (not just external_upload),
fixing the precedents_total that excluded 44 internal-committee records
- UI: courts table filters to source_type=court_ruling only;
committees table uses the new all_committees query
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add ability to mark case_law records as related (e.g. same appeal
through ועדת ערר → מנהלי → עליון):
- DB: case_law_relations join table (bidirectional, V11 migration)
- DB CRUD: add/remove/get_case_law_relations
- Service: get_precedent() now returns related_cases[]
- MCP: precedent_link_cases + precedent_unlink_cases tools
- REST: POST/DELETE /api/precedent-library/{id}/relations
- UI: RelatedCasesSection on detail page with search dialog and unlink
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Earlier commit afcc481 opened request_metadata_extraction and
request_halacha_extraction to all source kinds — but
list_pending_extraction_requests still hard-filtered to external_upload.
Result: stamping a queue request on an internal_committee row succeeded
silently, but the worker (and the queue badge) never saw it. Even with
the auto-wakeup added in c7132ba the CEO would wake, find 0 pending
items, and exit.
Drop the legacy filter so the queue listing matches the writer side.
Coolify deploy required for the FastAPI container to pick this up.
The "חלץ מטא-דאטה" / "חלץ הלכות" buttons in the UI were returning 404
for any precedent with `source_kind != 'external_upload'`. The original
restriction was meant to keep LLM extraction off internal-committee
imports (their metadata supposedly came from the case file system),
but the same precedent rows can still need re-extraction when ingest
produces broken data — e.g. the corrupted `subject_tags` value
`['[','"','ה','י',...]` that motivated this change (an early ingest
stored a JSON literal into a TEXT[] column, which Postgres split into
single chars).
Two changes here:
1. db.request_metadata_extraction / request_halacha_extraction:
drop the `AND source_kind='external_upload'` filter. The extractor
already preserves user values (only fills empty fields), so this
is safe.
2. precedent_metadata_extractor.extract_and_apply: detect the
character-by-character corruption above and treat it as empty so
the freshly-extracted tags actually replace the broken ones.
Heuristic: 3+ elements where every element is at most 2 chars
(legitimate tags are multi-character Hebrew words).
Coolify deploy required for the FastAPI container to pick this up.
Three-layer separation: style learning (style_corpus), appeals-committee decisions
(internal_committee), and court rulings (external_upload).
- SCHEMA_V10: chair_name + district columns on case_law and cases, partial indexes
- create_internal_committee_decision() DB upsert function
- search_precedent_library_semantic() now accepts source_kind/district/chair_name params
- search_precedent_library_hybrid() passes through new params
- services/internal_decisions.py: ingest_internal_decision, migrate_from_style_corpus,
migrate_from_external_corpus (identifies rows via source_type='appeals_committee')
- search_internal_decisions() MCP tool (server.py + tools/search.py)
- internal_decision_migrate() MCP admin tool
- Web endpoints: POST /api/internal-decisions/upload, POST /api/internal-decisions/migrate,
GET /api/internal-decisions
- ingest_final_version auto-ingests finalized decisions into internal corpus
- SKILL.md updated: agents now search internal + external in parallel, present separately
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When Paperclip wakes the CEO and the model issues an mcp__legal-ai__*
call within ~10s of session init, Claude Code sometimes returns
"No such tool available" because the legal-ai MCP server hasn't
finished bringing up its tool catalog yet. Observed twice today on
CMPA precedent-extraction wakeups (sessions 9989fbaf and a9c61801);
the agent fell back to bash + .venv/bin/python and finished the work,
but the race needed fixing on the server side.
Three changes that close the window:
1. Lazy schema init (services/db.py + server.py)
`init_schema()` was awaited inside the FastMCP lifespan, blocking
the `initialize`/`tools/list` handshake until ~10 CREATE TABLE IF
NOT EXISTS statements ran. Under contention (two CEOs waking at
once for different companies) this stretched. Now the lifespan
returns immediately and `get_pool()` runs the schema migrations
exactly once on first DB access, guarded by an asyncio.Lock.
tools/list is answered in milliseconds regardless of DB state.
2. Lazy heavy imports
- services/embeddings.py: voyageai (~450ms) loaded only inside
_get_client()
- services/extractor.py: google.cloud.vision (~550ms) loaded only
inside _get_vision_client() and _ocr_with_google_vision()
These two were being imported at module top from
legal_mcp.tools.documents -> services.processor -> services.{
extractor,embeddings}, so the FastMCP server couldn't even start
responding until both finished. Cold start dropped from 2.7s to
1.17s end-to-end (init + tools/list response).
3. Agent-side warmup + retry guidance (.claude/agents/legal-ceo.md)
Even with a fast server, the model can still race on the very
first call. The precedent-extraction section now tells the CEO
to call workflow_status as a warmup probe and to retry after a
short sleep if it sees "No such tool available", before falling
back to the python bypass.
Also expanded the precedent-tool whitelists on the sub-agents that
delegate halacha/library work (commits 4a9a6b7 + 7ee90dc added the
tools to the MCP server but only the CEO got them in its allowed
list). Added to: legal-researcher (full extraction set), legal-analyst
(library_get/list + halacha review), legal-writer (library lookups +
halacha_review), legal-qa (library_get + halacha_review), and the two
that the CEO was already missing (halacha_review, halachot_pending).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stage C: per-page image embeddings via voyage-multimodal-3 + hybrid
text+image search. Off by default; enable with MULTIMODAL_ENABLED=true.
- Schema V9: document_image_embeddings + precedent_image_embeddings
(vector(1024), page_number, image_thumbnail_path)
- extractor.render_pages_for_multimodal renders PDF pages at
MULTIMODAL_DPI (144) for embedding + JPEG thumbnails at
MULTIMODAL_THUMB_DPI (96) for UI preview, in one pass
- embeddings.embed_images calls voyage-multimodal-3 in 50-page batches
- services/hybrid_search.py orchestrator: rerank applied to text side
first (rerank-2 is text-only); image side cosine; weighted merge
with text_weight 0.65 (env-tunable); image-only pages surface as
match_type='image' so dense scanned content still appears
- processor.process_document and precedent_library.ingest_precedent
gated by flag — non-fatal on multimodal failure
- scripts/multimodal_backfill.py — idempotent per-case CLI to embed
existing documents without re-extracting text
Validated locally on a 5-page response brief: render 0.31s, embed 8.32s,
hybrid merge surfaces image rows correctly. Production rollout starts
with flag=false (no behavior change), then per-case A/B.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PATCH /api/halachot/{id} was returning 500 because the row included
``embedding`` as a numpy.ndarray of np.float32, which FastAPI's
jsonable_encoder cannot serialize (vars() and dict() both fail on it).
The bug had been latent — it triggered for the first time today after
the auto-approve batch left only low-confidence halachot for the chair
to review manually, and her first PATCH hit the unserializable response.
Replace ``RETURNING *`` with an explicit column list (everything except
``embedding``). Callers that need the embedding can re-fetch via
``get_halacha`` — but no current caller does.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Halachot extracted by halacha_extractor with confidence >= 0.80 are now
inserted with review_status='approved' instead of 'pending_review' —
they appear in search_precedent_library immediately. Halachot below the
threshold still require manual chair approval.
Threshold tunable via env (HALACHA_AUTO_APPROVE_THRESHOLD), defaults to
0.80. Rationale: 89% of historical extractions (356/400) score 0.80+,
spot-checks confirmed quality, and the manual review backlog was the
single biggest reason rerank-2 was returning passages-only on
ההבחנה-style queries.
After this change + the one-time backfill UPDATE, search now returns
9/10 halachot for "ההבחנה בין השבחה לפיצויים" instead of 0 — and the
top-3 are exact-match rules, not adjacent passages.
Reviewer field records "auto-approved (confidence ≥ X.XX)" with the
threshold value at insert time, for traceability.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
set_case_law_extraction_status and set_case_law_halacha_status now NULL
the corresponding *_requested_at timestamp when status transitions to
"completed" or "failed". Without this, completed rows kept lingering in
the local-MCP work queue (which scans by `WHERE *_requested_at IS NOT NULL`)
and the UI's isPrecedentActive check, leaving them undeletable until a
manual SQL cleanup.
The pre-existing process_pending_extractions path already called
clear_extraction_request, but other paths (re-extraction, status set
during upload) didn't — so the cleanup belongs at the status setter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
