The 4 'ambiguous' citation items flagged for chair turned out to be dead
orphan stubs: 0 inbound/outbound edges across all 5 citation mechanisms,
0 full_text, 0 halachot, 0 chunks/embeddings. A corpus-wide check found 15
such orphans total (incl. clean-looking ones). Per OpenCitations (keep an
id-less entity only if it is CITED — these are cited by nothing), these are
pure noise → deleted, not chair-judgment.
- 15 orphan cited_only stubs deleted (cited_only 46 -> 31); backup in
data/audit/fu2b-orphan-stub-cleanup-*.json.
- 0 malformed / 0 orphans remain; all 31 remaining stubs are cited.
- Combines with the 3 earlier mechanical normalizations. #70 fully done.
- Known forward-edge (no current data, no task): '+' combined-citation
handling in citation_extractor if it recurs in future extraction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A/B eval (eval_retrieval.py, 86-query gold-set) showed the 0.5 default was
mis-tuned: the image side was too heavy and dragged precedent_library recall
0.971 -> 0.885. Sweep 0.5..0.75 — at 0.65 multimodal beats text-only on every
overall metric AND every corpus (R@5 0.994 vs 0.989, nDCG@5 0.960 vs 0.944,
MRR 0.954 vs 0.936). Dafna approved.
- MULTIMODAL_TEXT_WEIGHT=0.65 set in Coolify (legal-ai, runtime) + redeploy.
- baseline.json updated to the 0.65 config (future regression reference).
- #15 done (premise was stale — multimodal already default on 110 docs; the
win was tuning the weight, not the backfill).
- #80 opened: the costly 140-doc legacy backfill is deferred until a targeted
image-answer gold-set proves the table/image value prop (untested here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A section that opens with a short header line ('דיון', 'טענות המשיבים')
followed by a paragraph larger than chunk_size flushed the header alone as a
tiny chunk. #55 added a query-time >=50 filter to hide these; this removes
them at the source.
_split_section: (1) don't flush a buffer still below MIN_CHUNK_CHARS — let it
absorb the next paragraph even if that overflows chunk_size, so a short header
rides with its following content; (2) fold a trailing tiny chunk back into its
predecessor.
Verified: re-chunked the 4 corpus docs that still had a tiny chunk
(ע"א 5138/04, בר"מ 2340/02, בג"ץ 6525/15, 403-17) — corpus-wide chunks<50
went 4 -> 0; all 4 stay embedded/searchable and rank top in a relevant search
(נווה שלום #1 for the s.19(ג)(1) exemption query). No regression.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Discovered closing #57: the current chunker still emits 4 tiny chunks that
are standalone section headings ('דיון', 'טענות המשיבים', ...). Low priority
— filtered at query time, search unaffected. Proposed fix: anchor a short
isolated heading forward into the following section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/rechunk_legacy_precedents.py: selects every case_law with a tiny
chunk (content<50 — the pre-fix chunker fingerprint) and runs
ingest.reindex_case_law (re-chunk+re-embed from stored full_text only, no
re-OCR/LLM, idempotent). Batch-idempotent (re-queries the affected set).
Run result (2026-06-03): 73 precedents reindexed, 0 failed. Tiny chunks
483 -> 4 (99.2%); total precedent_chunks 5019 -> 3115 (fragments merged).
Search verified healthy (substantial coherent passages, no errors).
The 4 residual tiny chunks are isolated section headings ('דיון',
'טענות המשיבים', ...) emitted by the CURRENT (fixed) chunker — not legacy
fragments — and are already filtered at query time (>=50, #55). Minor
chunker edge case, candidate #55 follow-up.
The DB chunk migration is already applied to prod; this commit is the script
+ SCRIPTS.md entry only (no app code change, no deploy needed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#78 (committee-upload wakeup) + #77 (case_number identity) shipped.
#76 (Paperclip create-task button): root-caused to ee=companyId guard —
button enabled on title only but submit requires a company; not safely
patchable via injection. Deferred with workaround + upstream note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pairs with the backend PR. Stops the citation (מראה-מקום) from being stored
as the identifier, and lets a wrong identifier be corrected after the fact.
- upload sheet: new required 'מספר תיק (מזהה ייחודי)' field for committee
decisions → sent as case_number; the citation field is now sent as the
separate citation (→ citation_formatted) instead of as case_number.
- edit sheet: the case_number block is now an editable input (was read-only).
Halachot/chunks key off case_law_id (UUID), so renaming case_number is safe.
- precedent-library.ts: InternalDecisionUploadInput += citation; PrecedentPatch
+= case_number.
- types.ts: regenerated (api:types) — PrecedentUpdateRequest now carries
case_number.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two identity fixes for the precedent corpus:
1. PrecedentUpdateRequest += case_number — the canonical identifier was not
in the edit model, so a wrong id captured at upload (e.g. the full
citation pasted into the field) could not be corrected. update_case_law
already whitelists case_number.
2. /api/internal-decisions/upload += citation form field — case_number is
now the clean identifier (e.g. 8027-25) and citation is the full
מראה-מקום, stored as citation_formatted up-front (previously the UI sent
the citation AS case_number, leaving the id polluted and citation_formatted
empty until extraction). Stored via a post-ingest update_case_law, not the
core INSERT.
Frontend (separate case_number field in the upload + edit sheets) follows in
a second PR after api:types regen.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The /api/internal-decisions/upload path (used by the UI for ועדת-ערר
decisions) never called pc_wake_for_precedent_extraction, so committee
decisions were stuck at halacha_extraction_status='pending' forever — the
CEO was never woken to drain the queue. Root cause behind 8027-25's stuck
extraction. The other two upload paths (precedent_library, missing-precedent)
already wake the CEO; this one was missing it.
- internal-decisions upload: add the wakeup, routing the company by case
number prefix (1xxx→רישוי, 8xxx→היטל, 9xxx→פיצויים) when practice_area is
empty (else an 8xxx case wrongly routes to the licensing CEO).
- all three call sites: the wake helper returns {ok:False} WITHOUT raising
on a skipped/failed wakeup; that was silently dropped. Now logged at
WARNING with the reason, and the upload progress carries extraction_queued.
Fallback drainer (scheduled precedent_process_pending) deferred — the
missing wakeup was the actual failure; manual precedent_process_pending
remains the recovery path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#34 don't manufacture doubt about unambiguous statutes (s.19(ג)(2));
#35 writer/QA two-sources-of-truth sync gap (DB vs drafts/decision.md).
Output of the weekly-feedback-analysis job, pending commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'חלץ עובדות שמאיות עכשיו' button returned HTTP 500. Root cause:
wake_analyst_for_appraiser_facts POSTs a child issue to Paperclip with
priority='normal', but Paperclip's ISSUE_PRIORITIES enum is only
critical|high|medium|low. createChildIssueSchema (Zod) rejects 'normal'
with 400 Bad Request; pc_request raise_for_status() turns it into a 500
surfaced to the chair. Fixed to 'medium' (the sole non-normal occurrence
in the repo).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Halachot carrying a corroboration badge (positive citation count or a
negative treatment) float to the top of 'הלכות שחולצו', ordered by
corroboration strength; the rest keep document order by halacha_index.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xhigh is the quality sweet-spot for a single precedent but very slow at scale
(64-chunk case ≈ 20 min). Bulk queue-drains (process_pending over many
precedents) now use a lighter effort to cut wall-clock; interactive single
re-extraction keeps xhigh quality.
- config.HALACHA_BULK_EXTRACT_EFFORT (env, default 'high'; set 'medium' for max
speed, 'xhigh' to match single).
- extract()/_extract_impl()/_extract_chunk() take an `effort` override threaded
to claude_session.query_json; None falls back to HALACHA_EXTRACT_EFFORT (xhigh).
- process_pending_extractions(kind='halacha') passes the bulk effort; single
reextract_halachot keeps xhigh.
Verified end-to-end (mocked LLM): _extract_chunk(effort='medium') → query_json
effort='medium'; effort=None → 'xhigh' fallback. Closes the open item in #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Halacha extraction held ALL chunk results in memory and stored once at the very
end — a crash/interrupt mid-run (e.g. the 2026-05-31 freeze) lost everything and
re-paid the full LLM cost on retry.
Now each chunk's halachot are stored AND the chunk is checkpointed
(precedent_chunks.halacha_extracted_at) the moment it finishes:
- V25 schema: precedent_chunks.halacha_extracted_at (per-chunk checkpoint).
- db.store_halachot_for_chunk: atomic per-chunk insert (halacha_index continues
from MAX, caller serializes via an in-process store-lock) + checkpoint mark.
- db.reset_halacha_extraction (force) / mark_all_chunks_extracted (legacy backfill).
- _extract_impl rewritten: resume by default (skip checkpointed chunks; failed
chunks stay pending and are retried; status stays 'processing' until all done);
force=True wipes + redoes all. reextract_halachot passes force=True; the queue
drain (process_pending) resumes by default.
- Legacy guard: a pre-V25 precedent (halachot exist, no checkpoints) is
backfilled and treated as complete — never re-extracted (would duplicate).
Verified on 9002-24 (55 halachot, legacy): resume → legacy-backfill, NO
duplication (stays 55), all chunks checkpointed. Index continuation: store at
55,56 after max 54, no collision. Tracks #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31: opus-4-8 @ xhigh extraction + overlapping driver processes (agent
fallback retries each spawn an independent `python -c` driver; process_pending is
serial WITHIN a process but the box ran 4-5 drivers in parallel) → 12-16 concurrent
xhigh `claude -p` procs → load 69 → hard reboot.
Fix: halacha_extractor.extract() now takes a Postgres advisory lock
(pg_try_advisory_lock, key 'HALA') before any work. If another extraction (any
process/agent/driver — all share the legal-ai DB) holds it, the call returns
status='busy' and the precedent stays pending for the next drain. Guarantees ONE
extraction at a time ACROSS PROCESSES — an in-process Semaphore cannot (drivers
are separate OS processes). Core logic moved to _extract_impl (unchanged) under
the lock. CHUNK_CONCURRENCY now env-tunable (HALACHA_CHUNK_CONCURRENCY, default 3).
Verified: while a lock is held, extract() returns 'busy' with no LLM call; lock
releases cleanly and the next extraction proceeds. Tracks #72.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
מסך העלאת הפסיקה הציג צ'קבוקס "הלכה מחייבת" עם ברירת מחדל true גם
להחלטות ועדת ערר (isCommittee), כך שהלכות שחולצו מהחלטה לא-מחייבת
תויגו rule_type='binding' — בסתירה להגדרה הדוקטרינרית (ועדת ערר =
persuasive בלבד, לא binding כמו עליון/מנהלי).
- מסלול ההגשה של החלטות ועדת ערר שולח כעת is_binding=false תמיד
- הצ'קבוקס ננעל (disabled+unchecked) כשזוהתה החלטת ועדת ערר, עם
הסבר שההלכות יסומנו persuasive
יישור דוקטרינרי בלבד — אין השפעה downstream על ranking/injection;
rule_type הוא תווית תצוגה, והשער הפונקציונלי הוא review_status.
TaskMaster #73
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
