Halachot carrying a corroboration badge (positive citation count or a
negative treatment) float to the top of 'הלכות שחולצו', ordered by
corroboration strength; the rest keep document order by halacha_index.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xhigh is the quality sweet-spot for a single precedent but very slow at scale
(64-chunk case ≈ 20 min). Bulk queue-drains (process_pending over many
precedents) now use a lighter effort to cut wall-clock; interactive single
re-extraction keeps xhigh quality.
- config.HALACHA_BULK_EXTRACT_EFFORT (env, default 'high'; set 'medium' for max
speed, 'xhigh' to match single).
- extract()/_extract_impl()/_extract_chunk() take an `effort` override threaded
to claude_session.query_json; None falls back to HALACHA_EXTRACT_EFFORT (xhigh).
- process_pending_extractions(kind='halacha') passes the bulk effort; single
reextract_halachot keeps xhigh.
Verified end-to-end (mocked LLM): _extract_chunk(effort='medium') → query_json
effort='medium'; effort=None → 'xhigh' fallback. Closes the open item in #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Halacha extraction held ALL chunk results in memory and stored once at the very
end — a crash/interrupt mid-run (e.g. the 2026-05-31 freeze) lost everything and
re-paid the full LLM cost on retry.
Now each chunk's halachot are stored AND the chunk is checkpointed
(precedent_chunks.halacha_extracted_at) the moment it finishes:
- V25 schema: precedent_chunks.halacha_extracted_at (per-chunk checkpoint).
- db.store_halachot_for_chunk: atomic per-chunk insert (halacha_index continues
from MAX, caller serializes via an in-process store-lock) + checkpoint mark.
- db.reset_halacha_extraction (force) / mark_all_chunks_extracted (legacy backfill).
- _extract_impl rewritten: resume by default (skip checkpointed chunks; failed
chunks stay pending and are retried; status stays 'processing' until all done);
force=True wipes + redoes all. reextract_halachot passes force=True; the queue
drain (process_pending) resumes by default.
- Legacy guard: a pre-V25 precedent (halachot exist, no checkpoints) is
backfilled and treated as complete — never re-extracted (would duplicate).
Verified on 9002-24 (55 halachot, legacy): resume → legacy-backfill, NO
duplication (stays 55), all chunks checkpointed. Index continuation: store at
55,56 after max 54, no collision. Tracks #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31: opus-4-8 @ xhigh extraction + overlapping driver processes (agent
fallback retries each spawn an independent `python -c` driver; process_pending is
serial WITHIN a process but the box ran 4-5 drivers in parallel) → 12-16 concurrent
xhigh `claude -p` procs → load 69 → hard reboot.
Fix: halacha_extractor.extract() now takes a Postgres advisory lock
(pg_try_advisory_lock, key 'HALA') before any work. If another extraction (any
process/agent/driver — all share the legal-ai DB) holds it, the call returns
status='busy' and the precedent stays pending for the next drain. Guarantees ONE
extraction at a time ACROSS PROCESSES — an in-process Semaphore cannot (drivers
are separate OS processes). Core logic moved to _extract_impl (unchanged) under
the lock. CHUNK_CONCURRENCY now env-tunable (HALACHA_CHUNK_CONCURRENCY, default 3).
Verified: while a lock is held, extract() returns 'busy' with no LLM call; lock
releases cleanly and the next extraction proceeds. Tracks #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
מסך העלאת הפסיקה הציג צ'קבוקס "הלכה מחייבת" עם ברירת מחדל true גם
להחלטות ועדת ערר (isCommittee), כך שהלכות שחולצו מהחלטה לא-מחייבת
תויגו rule_type='binding' — בסתירה להגדרה הדוקטרינרית (ועדת ערר =
persuasive בלבד, לא binding כמו עליון/מנהלי).
- מסלול ההגשה של החלטות ועדת ערר שולח כעת is_binding=false תמיד
- הצ'קבוקס ננעל (disabled+unchecked) כשזוהתה החלטת ועדת ערר, עם
הסבר שההלכות יסומנו persuasive
יישור דוקטרינרי בלבד — אין השפעה downstream על ranking/injection;
rule_type הוא תווית תצוגה, והשער הפונקציונלי הוא review_status.
TaskMaster #73
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnosis from the FU-5 eval: co-relevant precedents for broad legal questions
rank 15-16 (retrieved, not absent — recall ~1.0 by rank 20). Tracked as a
deliberate, harness-measured tuning task rather than an unmeasured global limit
change (which affects UI + writer agents + token cost).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
חיווט ספ-המערכת לסוכני-Paperclip כך שכל סוכן חייב לקרוא את 00-constitution
תחילה, ואז את ספ-התחום הרלוונטי לתפקידו (לפי טבלת X4 §2) — לפני עבודה מהותית.
- HEARTBEAT.md: סעיף עליון "קריאת-ספ — קודם החוקה (00), אז ספ-התחום" לפני §0–§8,
עם טבלת תפקיד→ספ ל-8 הסוכנים.
- 8 קבצי-סוכן (ceo/proofreader/researcher/analyst/writer/qa/exporter/hermes):
סעיף "קרא לפני פעולה (INV-AG1)" בראש הגוף.
- X4-agents.md: שדה "אכיפה" של INV-AG1 → "מחוּוט (פרוצדורלי)"; §5 → "בוצע".
אכיפה פרוצדורלית בכוונה — invariant פרויקטלי-תפעולי, אין שער-קוד שמכריח קריאה.
prereq לסוכני-התהליך (תת-פרויקט 5). gap-audit נשמר כ-snapshot (כמו FU-8a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gold-set was 77 known-item probes (query=case_name). Added 9 chair-approved
SEMANTIC queries (S1–S9) — a real legal question per row, relevant = the
precedents that should surface (drawn from subject_tags, chair-confirmed). These
test what matters: does retrieval answer a legal issue, not just find a case by
name. source='chair' (preserved across re-bootstrap). practice_area left empty
so the filter never excludes a cross-tagged precedent (s.197 rulings sit under
betterment_levy).
Baseline now 86 queries. Finding from the 9 semantic queries: MRR ≈ 1.0 — the
system surfaces a lead relevant precedent at rank 1 for nearly every question —
but R@10 ranges 0.5–1.0: for broad questions with many co-relevant precedents
(e.g. נטרול תמ"א 38 = 5 relevant → R@10 0.60; שמאי מכריע = 2 → 0.50) some
co-relevant rulings miss the top-10. Lead-precedent retrieval is strong;
exhaustive multi-precedent recall is the gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chair review of the FU-5 gold-set surfaced one internal_committee record whose
case_name was a code ("ARAR-24-9002") rather than a real name. Per the chair's
citation (ערר 9002/24 קרקעות ירושלים 2 בע"מ נ' הוועדה המקומית ירושלים, נבו
13.8.2025, a s.197 compensation appeal), case_name corrected in the DB to
"קרקעות ירושלים 2" (case_number 9002-24 and citation_formatted were already
correct; only 1 such code-named record exists corpus-wide). Re-bootstrapped the
gold-set (the known-item query is now the real name) and refreshed baseline
(aggregate unchanged — the case retrieves identically under the corrected name).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gold-set card read data/eval/gold-set.jsonl, but .dockerignore excludes
data/ from the build context, so the file is never in the container and the
card silently never rendered. Baking eval data into the image is the wrong
layering (data/ is runtime volumes). The gold-set review is a one-time task,
not a recurring chair queue, so it doesn't belong on the live board — it's
tracked via task #63 and reviewed directly with the chair. The board now
returns the 4 robust DB-backed gates (halachot, missing precedents, feedback,
qa_failed). Removes the best-effort file read + its unused Path import.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dafna asked for a single page under the prod site listing everything she needs
to approve, so nothing is forgotten — the visible embodiment of INV-G10 (human
gates) and INV-QA1 (halacha backlog must be visible).
Backend — GET /api/chair/pending aggregates every pending chair gate, each as a
direct source query (count + sample + action link):
- halachot review backlog (review_status='pending_review') + oldest
- open missing precedents
- unresolved chair_feedback
- qa_failed cases
- gold-set review (FU-5, file-based, best-effort: total vs source='chair')
Frontend — /approvals page ("מרכז אישורים"):
- src/lib/api/chair.ts — usePendingApprovals() (hand-typed until next api:types)
- src/app/approvals/page.tsx — card per category, severity-coloured count, sample
rows, oldest-pending date, link to where each is handled; live (60s refetch)
- app-shell nav: "מרכז אישורים" in the work group + total-pending badge (quiet at 0)
Live counts at build time surfaced the value immediately: 226 open missing
precedents, 178 pending halachot, 20 unapplied feedback notes, 1 qa_failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers GAP-11 (INV-RET4/G8) and GAP-14 (INV-QA1/G10). Retrieval quality was
never measured (only telemetry observation) and the halacha review backlog was
invisible (the 10/19 gap was found by accident).
Unit B — backlog visibility (pure code, container):
- metrics.halacha_backlog(conn) → {pending_review, approved, rejected, published,
total, oldest_pending_at}; surfaced in metrics.get_dashboard() (get_metrics MCP
tool) and /api/system/diagnostics. Live count revealed 178 pending / 1552 total,
oldest from 2026-05-03 — previously invisible.
Unit A — retrieval eval harness (host-side scripts):
- scripts/eval_gold_bootstrap.py — seeds data/eval/gold-set.jsonl. Two sources:
citations (cited==relevant via search_relevance_feedback — empty until decisions
cite precedents) and known_item (query=case_name → relevant=self; a real
citation-free signal, the methodology #52 checked by hand). Idempotent; preserves
source='chair' rows.
- scripts/eval_retrieval.py — runs the production retrieval path (search_library /
search_internal) over the gold-set; computes precision@k, recall@k, MRR, nDCG@k
(k=5,10); aggregates overall + per-corpus + per-practice_area; writes a report and
a delta vs committed baseline.json (which records the retrieval_config it reflects).
--self-test unit-checks the metric math offline.
Gold-set strategy = hybrid (chair decision): bootstrap + chair review. The citation
source is empty today (0 cited precedents in decisions), so the seed is known-item
(77 queries: 54 internal_decisions + 23 precedent_library). The gold-set is
PROVISIONAL until Dafna reviews it (the domain chair-gate).
Baseline (production config: multimodal+rerank on): R@10=0.987, MRR=0.837,
nDCG@10=0.872. Finding: MULTIMODAL_ENABLED=true slightly lowers known-item recall
(image-page results displace exact name matches) — relevant to #15. precedent_library
weaker than internal (R@10 0.957 vs 1.0) — one external precedent unfindable by name.
"CI gate" realized as discipline (re-runnable harness + committed baseline + run
before/after any retrieval-layer change) — retrieval needs prod DB + Voyage, no CI
runner has that access.
Spec: docs/superpowers/specs/2026-05-31-fu5-eval-harness-design.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
