Reported: an agent claimed the case had no documents because document_list
returned empty — but the documents exist. Root cause: get_case_by_number did
an exact `WHERE case_number = $1`, so any formatting variant of the number
silently failed to resolve. Verified on 8137-24 (9 docs): "8137/24",
"ערר 8137-24", leading/trailing space, and "בל\"מ 8126/03/25" all returned
"תיק לא נמצא", which the agent read as "no documents" and went blind.
Add _normalize_case_number (strip leading proceeding-type prefix to the first
digit, trim, unify '/'→'-') and a normalized fallback in the lookup query
(exact match preferred via ORDER BY). One fix covers every case_number-scoped
tool (document_list, extract_references, search_case_documents, get_claims,
drafting, ...). Bogus numbers still correctly resolve to "not found". (#58)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of "agent can't find the Agasi decision in the corpus" (CMPA-55):
the decision was fully ingested, but the retrieval layer failed on the
realistic agent query — searching by case name.
- RC-A (#52): lexical tsvector covered only chunk content + halacha text,
so a bare-name query ("אגסי") matched decisions that *cite* the case, not
the case itself. Add meta_tsv on case_law(case_name, case_number) (SCHEMA
V20) and OR it into the lexical halacha/chunk SQL with a match boost, so a
name/number hit surfaces the case's own rows. Agasi: rank 4 → rank 1.
- RC-B (#53): precedent_library_list hard-defaulted source_kind=external_upload
and never exposed the param, hiding uploaded ערר/בל"מ (internal_committee)
decisions. Thread source_kind through service → tool → MCP tool (supports
'internal_committee' / 'all_committees').
- #54: agent instructions (researcher/analyst/writer) — search-by-name
protocol: add content/case-number, search both corpora, use all_committees
before declaring "not in corpus".
- #55: chunker produced tiny fragment chunks ("דיון", "החלטה") from header
keywords matched mid-sentence. Anchor SECTION_PATTERNS to line start +
merge sub-min sections; exclude <50-char fragments at query time (484
existing fragments hidden; full re-chunk tracked as #57).
Tests: scripts/test_retrieval_by_name.py (name ranks case above citer +
substantive regressions); chunker unit checks (0 tiny chunks). New findings
filed as tasks #56 (halacha source_kind leak) and #57 (re-chunk migration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address security-review finding: the host-side legal-chat-service was
binding 0.0.0.0:8770 with no authentication. The service spawns the
claude CLI, whose tool set includes Bash + Edit — so an unauthenticated
/chat/start is effectively RCE. Oracle Cloud's security list closes the
port externally, but defense-in-depth requires two independent layers:
1. Bind defaults to 10.0.1.1 (docker0 bridge gateway). Reachable from
containers on docker bridges (the legal-ai container has a route via
the coolify network), invisible to anything outside the host. The
--host flag is still configurable for local-dev (127.0.0.1) or
special-case deployments, but 0.0.0.0 is explicitly discouraged in
the docstring.
2. /chat/start requires Authorization: Bearer <LEGAL_CHAT_SHARED_SECRET>.
The secret is loaded from /home/chaim/.legal-chat-service.env (chmod
600, off-repo) by the pm2 ecosystem and mirrored as a Coolify env
var so the FastAPI chat_proxy sends a matching header. hmac.compare_digest
prevents timing oracles. /health stays unauthenticated (static OK,
no subprocess) so the FastAPI proxy can probe liveness without the
secret.
The service refuses to start if LEGAL_CHAT_SHARED_SECRET is empty or
shorter than 24 chars — no silent fallback to an open mode.
When the Infisical MCP comes back, migrate the secret into the vault
at /_GUIDELINES per the project secrets policy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six-phase upgrade of /training from a read-only dashboard into a full
Style Studio for managing Daphna's style corpus.
- Upload Sheet on /training: file → proofread preview → commit (no more
CLI-only `upload-training` skill).
- Rich corpus metadata: GET /api/training/corpus returns summary, outcome,
key_principles, page_count, parties (regex), legal_citation, lessons_count.
PATCH endpoint for chair edits. CorpusDetailDrawer with 4 tabs (details
/content/lessons/patterns) replaces the bare table row.
- LLM metadata enrichment: style_metadata_extractor + MCP tools
(style_corpus_enrich, style_corpus_pending_enrichment) fill summary
/outcome/key_principles via claude_session (free, host-side).
- Per-decision lessons: new decision_lessons table + 4 REST endpoints +
LessonsTab in drawer; hermes-curator now auto-posts findings as
decision_lessons(source=curator).
- Curator Portrait tab: prompt rendered with link to Gitea, recent
curator findings, style_analyzer training prompts, propose-change
form that writes proposals to data/curator-proposals/ for manual
chair review (no auto-mutation of the agent file).
- Style chat tab: SSE-streamed conversations with the style agent.
New host-side pm2 service (legal-chat-service, port 8770) wraps
claude CLI with stream-json + --resume continuation; FastAPI proxies
via host.docker.internal. Zero API cost — uses chaim's claude.ai
subscription. chat_conversations + chat_messages persist history.
Architecture: keeps the existing rule that claude_session only runs
on the host (not the container). The new legal-chat-service is the
canonical bridge between the container and the local CLI for the chat
feature; everything else (upload, metadata, lessons) stays within the
container's existing capabilities.
Audit script (scripts/audit_training_corpus.py) included for verifying
which corpus rows still need enrichment.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Until now, "case_number" was the only stored identifier for a precedent.
But a *citation per the Israeli unified citation rules* is a different
beast — it has bold parties, an unbold prefix (court abbrev + panel/
district parenthetical + case number), and an unbold trailing reporter
(נבו / פ"ד...). Without storing it as a first-class field we couldn't
hand the chair a one-click "copy as citation" experience for pasting
into decisions.
Changes:
- Schema V19: case_law.citation_formatted TEXT (Markdown — parties
wrapped in **…** so the copy helper can render <strong> for Word/Docs
paste and keep plain-text fallback meaningful).
- Metadata extractor: composes citation_formatted from the document
text per the unified citation rules, with worked examples for ע"א /
עת"מ / ערר / בל"מ in the prompt. Refuses to store half-formed strings.
- PATCH /api/precedent-library/{id} accepts citation_formatted so the
chair can correct LLM mistakes.
- /precedents/[id]: dedicated "מראה מקום" block with bold rendering,
a copy-to-clipboard button (text/html + text/plain so Word keeps
the bolds), and an inline edit textarea.
- /precedents list rows: link displays the formatted citation when
available, with a small inline copy button — falls back to the bare
case_number for older rows.
Backfill of existing rows happens by re-stamping the extraction queue
once V19 has rolled out and the new field is reachable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The missing-precedents drawer + general precedent upload both required
the user to type chair_name, district, practice_area, court, date etc.
upfront — even though those fields can be (and already are, post-upload)
extracted from the document text by the LLM. The metadata-extraction
wakeup also only fired for the /precedent-library/upload path, leaving
missing-precedents committee uploads stuck with whatever stub the user
typed.
Changes:
- Extractor learns chair_name + district, overwrites the new
PLACEHOLDER_PENDING_EXTRACTION sentinel for internal_committee rows
(the DB CHECK forces non-empty; we stamp the placeholder at insert).
- missing_precedent_upload no longer 400s on missing chair/district;
it infers district from the citation when possible, falls back to
the placeholder, and always fires pc_wake_for_precedent_extraction
so the LLM can fill in the rest.
- Both upload sheets default to file (+ citation) only; every other
field is tucked into a closed <details> labeled "אופציונלי — דריסה
ידנית של שדות שיחולצו אוטומטית". Required validators on chair/
district/practice_area dropped — the LLM fills them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "חלץ עובדות שמאיות" UI button hit POST /api/cases/{n}/extract-appraiser-facts
which called appraiser_facts_extractor inline — that shells out to the local
`claude` CLI, which is absent in the Coolify container, so every doc errored,
the per-doc try/except swallowed it, and the response was "completed, 0 facts".
Refactored the endpoint to wake the legal-analyst of the correct company via
Paperclip (same pattern as wake_curator_for_final), and surface
extraction_failed instead of "completed" when every doc errored.
## #34 — Daphna's internal citation graph
New schema V16 (V15 was already used by proceeding_type): table
``precedent_internal_citations`` (source→cited, with cited_case_law_id
nullable for citations whose target isn't in the corpus yet) + 3
indexes (source, target, unlinked).
New service ``citation_extractor.py`` with regex patterns for ערר /
בל"מ / עע"מ / בר"מ / עמ"נ / ע"א / בג"ץ / רע"א — accepts both ``\/``
and ``-`` separators, requires actual parenthesized district label
to avoid greedy mid-paragraph captures. Resolves citations against
``case_law.case_number`` substring; default confidence 0.90 linked,
0.75 unlinked. ON CONFLICT DO NOTHING on (source, cited_case_number).
3 new MCP tools: ``extract_internal_citations``,
``list_internal_citations``, ``list_incoming_citations``. Optional
flag ``include_cited_by=True`` on ``search_internal_decisions``
appends cited-by candidates as ``match_type='cited_by'`` stubs.
Bulk-extracted from 40 internal_committee rows authored by דפנה תמיר:
**353 distinct citations, 348 stored, 96 linked / 252 unlinked**.
Top citers: 1079/24 (30), 1024/24 (19), 1009/25 (18). Top unlinked
target: ע"א 3213/97 (cited 5x) — natural #35 candidates.
## #32 — Wide-modal precedent edit
`precedent-edit-sheet.tsx`: ``<Sheet side="left">`` → centered
``<Dialog>`` with ``sm:max-w-4xl`` ``max-h-[90vh]`` ``overflow-y-auto``.
Component API unchanged so existing callers
(`/precedents/[id]/page.tsx`, `library-list-panel.tsx`) work as-is.
RTL preserved. Mobile falls back to near-full-width via shadcn default.
## #13 — 403/17 verification
`case_law e151fc25-...` (אהרון ברק - תכנית רחביה) already in perfect
shape after Stage A work: all metadata fields populated, 351 halachot
with avg_conf=0.864 (well above 0.78 threshold). No re-extraction
needed; closing task as verified.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Same case_number can exist as both a regular appeal (ערר) and an
extension-of-time request (בל"מ), and we were inferring the difference
from appeal_subtype prefixes — fragile, and case-number lookups
weren't disambiguated. Now stored as a first-class field on both
case_law (corpus) and cases (live cases), with partial unique indexes
on (case_number, proceeding_type).
- SCHEMA_V15: column + CHECK constraints + backfill from
appeal_subtype LIKE 'extension_request_%' + partial unique indexes
replace the old global UNIQUE(case_number).
- derive_proceeding_type() centralizes the inference rule
(extension_request_* → בל"מ; subject regex fallback; default ערר).
- Metadata extractor prompt asks Claude to populate the new field
explicitly; apply_to_record writes it for internal_committee rows.
- internal_decision_upload, case_create, case_update accept an
optional proceeding_type; FastAPI request models expose it.
- Wizard + edit dialog get a sided Select; case header renders the
resolved label (ערר / בל"מ).
- Uploaded the 2 staged בל"מ decisions on betterment levy:
8126/24 (סופר נוח, 13 chunks), 8047/23 (הרנון, 48 chunks).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four parallel sub-agents closed the remaining critical gaps from the
26/05 Stage A/B sprint. Each block independently tested; aggregated here.
## #30/#31 finalizers (sub-agent A)
* Auto-derive practice_area in case_create from case_number prefix
(1xxx→rishuy_uvniya, 8xxx→betterment_levy, 9xxx→compensation_197);
default for CaseCreateRequest is now "" (the DB constraint catches
any stray "appeals_committee").
* practice_area.py: derive_subtype now handles axis-B domain values
(rishuy_uvniya/betterment_levy/compensation_197) without parsing the
case number; new helper derive_domain_practice_area().
* Halacha re-extraction verified unnecessary — all 6 reclassified
records already had is_binding=false and approved halachot.
* Regression tests: 6 cases in tests/test_corpus_constraints.py
covering practice_area enum, internal-committee chair/district,
external-upload arar prefix, MCP guard.
* UI: district input → Select dropdown (7 districts) in
precedent-edit-sheet.tsx, preserving legacy free-text values.
## #37 בל"מ subtypes (sub-agent B)
* 3 new appeal_subtypes: extension_request_{building_permit,
betterment_levy,compensation}. APPEALS_COMMITTEE_SUBTYPES extended,
SUBTYPES_BY_AREA mappings added.
* New helpers: is_blam_subject(), is_blam_subtype(),
derive_subtype_with_blam(case_number, subject, practice_area).
case_create now uses it to auto-detect "בקשה להארכת מועד" subjects.
* 3 methodology templates under docs/methodology/extension-request-*.md.
* paperclip_client.py mapping updated for the 3 new subtypes
(extension_request_building_permit→CMP, the other two→CMPA).
* Frontend: bilingual "בל"מ" badge + filter dropdown on cases list +
detail header; appeal-type-bars collapseBlam() merges בל"מ into its
parent domain for aggregate bars.
* Wizard auto-detects בל"מ from subject during case creation.
* 3 Berlinger cases (1017/1018/1019-03-26) migrated to
appeal_subtype=extension_request_building_permit via psql.
## #35 missing_precedents feature (sub-agent C)
* Schema V13: missing_precedents table (citation, case_id, party,
legal_topic, status, linked_case_law_id, claim_quote, ...) +
FK constraints + 3 indexes. Applied via psql + idempotent migration.
* 6 db.py service functions, 3 MCP tools, 6 FastAPI endpoints
(POST/GET/PATCH/DELETE/upload — upload routes by citation prefix
to ingest_internal_decision or ingest_precedent).
* Next.js page /missing-precedents with 5 status tabs + filters +
sidebar badge counter + detail drawer with metadata edit + smart
upload form that switches fields per committee/court.
* Bootstrap: 7 rows imported from the JSON file
(3 citations × cases, all status=closed with linked_case_law_id).
* legal-researcher.md: new §2ב.5 with missing_precedent_create
usage + dedup semantics + tool grant.
## #36 legal_arguments aggregation (sub-agent D)
* Schema V14: legal_arguments + legal_argument_propositions M:M.
Applied via psql.
* New service argument_aggregator.py with two functions —
aggregate_claims_to_arguments() (Claude CLI / claude_session) and
get_legal_arguments(). Graceful llm_unavailable handling when CLI
is missing (containers).
* 2 MCP tools + 2 API endpoints (POST .../aggregate-arguments as
BackgroundTask, GET .../legal-arguments).
* Frontend: shadcn Accordion + new legal-arguments-panel.tsx with
hierarchical (party → priority badge → arguments) display, "טיעונים"
tab on the case page, "חשב/חשב מחדש" buttons.
* scripts/backfill_legal_arguments.py + SCRIPTS.md entry — dry-run
found 8 candidate cases including 1017/1018/1019.
## Open follow-ups (intentionally deferred)
* npm run api:types in web-ui (CLAUDE.md flow) — recommended before
the next UI commit; not required for backend deployment.
* Run backfill_legal_arguments.py --apply once the container picks up
the new aggregator service.
* webhook on missing-precedents upload-close to Paperclip (optional).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five enhancements to the precedent retrieval stack:
* **#44 HNSW indexes** for precedent_chunks + halachot (replacing IVFFlat
lists=50). Build time ~3s combined. Better recall@10 with pgvector 0.8.2.
* **#45 Halacha sweep** — 96 pending halachot at conf>=0.78 promoted to
approved (1141 → 1237). Cluster at conf=0.78 spot-checked OK. Applied
via psql only — env HALACHA_AUTO_APPROVE_THRESHOLD unchanged (0.80).
* **#43 MMR diversity** — search_precedent_library_hybrid now caps at
``max_per_case_law=2`` (default). Prevents one precedent dominating
top-10 when many of its chunks/halachot rank high. New helper
``_diversify_by_case_law`` in hybrid_search.py.
* **#46 Dynamic halacha boost** — replaces the static ``score+=0.05``
with ``score+=confidence*0.06``. Calibrated so avg-confidence (~0.85)
stays at +0.05; high-conf halachot get a slight extra lift, low-conf
ones get less. Behaviour preserved at the mean.
* **#41 BM25/tsvector hybrid + RRF**. Schema V12 adds STORED tsvector
columns ``precedent_chunks.content_tsv`` and ``halachot.rule_tsv``
(using simple config — Postgres has no Hebrew stemmer) + GIN indexes.
New ``db.search_precedent_library_lexical`` mirrors the semantic
function with ts_rank_cd over plainto_tsquery. ``hybrid_search``
runs sem+lex in parallel and fuses via RRF before rerank. Toggle:
env ``BM25_HYBRID_ENABLED`` (default true), graceful fallback to
semantic-only on lexical failure.
#40 (VOYAGE_RERANK_ENABLED) was already true in Coolify env; no change.
#42 (Claude Haiku query expansion) deferred — latency + cost concerns
warrant a separate plan; the bm25 lexical leg already recovers most of
the exact-string recall #42 was meant to address.
Closes TaskMaster #41, #43-#46.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Defense in depth — the MCP wrapper guard catches researcher uploads, but
the HTTP API (/api/precedent-library/upload) bypasses the wrapper and
calls services.precedent_library.ingest_precedent directly. The guard
now also lives in the service, so HTTP uploads of ערר/בל"מ citations
to the external corpus get rejected at the source.
Companion to DB constraint case_law_external_arar_check (applied via
psql) — three independent layers now enforce the same invariant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Born-digital Hebrew PDFs from legal software often encode gershayim (״)
as double-yod (יי), producing the same corruption patterns as OCR.
The fixer was only called after Google Cloud Vision OCR — digitally
created PDFs that passed quality checks received no correction.
Changes:
- Apply _fix_hebrew_quotes() in the direct extraction path
- Add 'בליימ' → 'בל"מ' (בקשה להארכת מועד — systematic corruption in 1017-03-26)
- Add 'תמייא' → 'תמ"א' (תכנית מתאר ארצית)
- Update docstring to reflect the broader scope
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Wrap date.fromisoformat() in try/except in case_update tool — prevents
unhandled ValueError from surfacing as 500; FastAPI now catches it as 422
- Add DialogDescription (sr-only) to 5 dialogs missing aria-describedby:
documents-panel preview + delete, drafts-panel delete + feedback, link-related-dialog
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fields existed in DB and Precedent type but were missing from:
- PrecedentUpdateRequest (backend model)
- update_case_law allowed set (db layer)
- PrecedentPatch (frontend type)
- precedent-edit-sheet form state, inputs, and patch payload
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The legal-analyst agent was generating a longer placeholder form
[ימולא ע"י יו"ר הוועדה — עמדה/הנחיה לגבי סוגיה זו שתשמש את סוכן הכתיבה]
which _is_placeholder() did not match (substring check fails because ] is
further along in the longer form). Result: UI showed "✓ עמדה נקבעה" (green)
for all 4 issues even though no chair direction had been entered.
Fixes:
1. research_md.py: add regex fallback — any text starting with [ימולא is a placeholder
2. legal-analyst.md: template now emits the standard short placeholder only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CEO wakes for weekly-feedback-job via agents.invoke without issueId,
so $PAPERCLIP_TASK_ID is empty. Removed steps 4-5 (comment + close
issue) from handler — now file-write only with stdout logging.
Also commits pending docs and agent instructions from prior session.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If pc_wake_ceo fails, the endpoint now raises HTTP 502 and skips the
case_update to processing — preventing cases from silently getting stuck
with no CEO running. Also adds `processing` to CEO routing table and
updates case_list docstring with full status list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix keyboard navigation bug: React was reusing the submit button DOM element
when transitioning "הבא" → "צור תיק", retaining focus and causing Enter to
auto-submit step 3. Added key props to force element replacement.
- CaseEditDialog now covers all wizard fields: appellants, respondents,
property_address, permit_number (in addition to existing title, subject,
hearing_date, expected_outcome, notes).
- When case title changes, Paperclip project name is updated in background
via new update_project_name() in paperclip_client.py.
- Extended CaseUpdateRequest, case_update MCP tool, and caseUpdateSchema
to carry the new fields end-to-end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pass req.value directly to asyncpg instead of json.dumps(req.value).
When a Python string was passed with ::jsonb, asyncpg encoded it as a
JSONB string (not an array), causing the frontend spread operator to
split it into individual characters — one textarea per character.
Also fix typo in DISCUSSION_RULES default: "אסה" → "מאסה".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs caused all 5 interim blocks to fail with "Claude CLI failed
(exit 1): unknown error":
1. source_context was embedded BOTH inside the prompt template (via
{source_context}) AND prepended again in write_block — doubling every
block's context size (232K chars × 2 = 465K chars).
2. _build_source_context loaded all 9 case documents for every block
regardless of relevance.
Fixes:
- Remove the duplicate source_context prepend in write_block; the
template already contains it via {source_context}
- Add per-block document filtering (_BLOCK_DOC_TYPES): block-he/zayin →
empty, block-chet → protocol only, block-tet → appraisals only
- Add 400K char guard before calling claude -p with a descriptive error
(vs opaque "exit 1: unknown error")
- Add prompt-size warning and size info in claude_session error messages
Result: block-he 0 chars, block-zayin 0 chars, block-vav ~172K,
block-chet ~45K, block-tet ~300K (all under 400K limit)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- DB: add 'all_committees' virtual source_kind covering internal_committee
+ external_upload appeals_committee rows in one query
- DB: stats now count all case_law rows (not just external_upload),
fixing the precedents_total that excluded 44 internal-committee records
- UI: courts table filters to source_type=court_ruling only;
committees table uses the new all_committees query
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add ability to mark case_law records as related (e.g. same appeal
through ועדת ערר → מנהלי → עליון):
- DB: case_law_relations join table (bidirectional, V11 migration)
- DB CRUD: add/remove/get_case_law_relations
- Service: get_precedent() now returns related_cases[]
- MCP: precedent_link_cases + precedent_unlink_cases tools
- REST: POST/DELETE /api/precedent-library/{id}/relations
- UI: RelatedCasesSection on detail page with search dialog and unlink
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
reextract_metadata / reextract_halachot extract & apply but never cleared
metadata_extraction_requested_at / halacha_extraction_requested_at —
only the bulk worker (process_pending_extractions) did. Result: clicking
"חלץ מטא-דאטה" on the edit sheet (or calling precedent_extract_metadata
directly) left the row stuck in the queue forever, with the UI badge
showing "ממתין לחילוץ" even after extraction succeeded.
Mirror the worker's behaviour: on success ('completed' / 'no_changes' /
'no_halachot'), call db.clear_extraction_request to drain the queue.
Coolify deploy required for the FastAPI container; local MCP server
needs a process restart for the change to take effect (long-running).
Earlier commit afcc481 opened request_metadata_extraction and
request_halacha_extraction to all source kinds — but
list_pending_extraction_requests still hard-filtered to external_upload.
Result: stamping a queue request on an internal_committee row succeeded
silently, but the worker (and the queue badge) never saw it. Even with
the auto-wakeup added in c7132ba the CEO would wake, find 0 pending
items, and exit.
Drop the legacy filter so the queue listing matches the writer side.
Coolify deploy required for the FastAPI container to pick this up.
The "חלץ מטא-דאטה" / "חלץ הלכות" buttons in the UI were returning 404
for any precedent with `source_kind != 'external_upload'`. The original
restriction was meant to keep LLM extraction off internal-committee
imports (their metadata supposedly came from the case file system),
but the same precedent rows can still need re-extraction when ingest
produces broken data — e.g. the corrupted `subject_tags` value
`['[','"','ה','י',...]` that motivated this change (an early ingest
stored a JSON literal into a TEXT[] column, which Postgres split into
single chars).
Two changes here:
1. db.request_metadata_extraction / request_halacha_extraction:
drop the `AND source_kind='external_upload'` filter. The extractor
already preserves user values (only fills empty fields), so this
is safe.
2. precedent_metadata_extractor.extract_and_apply: detect the
character-by-character corruption above and treat it as empty so
the freshly-extracted tags actually replace the broken ones.
Heuristic: 3+ elements where every element is at most 2 chars
(legitimate tags are multi-character Hebrew words).
Coolify deploy required for the FastAPI container to pick this up.
The Hermes Knowledge Curator's hermes-curator.md says it must be able to
read both DOCX and PDF final decisions. The original implementation
hardcoded the .docx extension only. Extend to try .docx → .pdf → .doc →
.rtf → .txt → .md, returning the first match. extractor.extract_text
already supports all six formats, so no extractor changes needed.
If none found, the not_found response now includes the tried_extensions
list so the caller knows what was attempted.
Verified on case 1130-25 (.docx still picked first) and tested via
`curator-cmp mcp test legal-ai`.
The Knowledge Curator (Hermes) couldn't read סופי-{case}.docx because
document_get_text only works on rows in the documents table — the final
file is just a copy in the case's exports/ directory, not a tracked
document. CMP-71 hit this and produced an unproductive interaction
asking the user how to fix the access issue.
Add a new MCP tool that:
- Locates exports/סופי-{case_number}.docx via config.find_case_dir
- Extracts text using the existing extractor service (python-docx based)
- Returns JSON with status + text + page_count + truncation info
- Optional max_chars cap for large decisions
Smoke test on case 1130-25: 400-char preview returns proper Hebrew text
beginning with "לפנינו ערר על החלטת הוועדה המקומית...".
The local MCP server reloads on next Hermes spawn (stdio mode), so the
tool is immediately available — no Coolify deploy needed.
Curator's promptTemplate (DB-stored) updated to use the new tool as the
primary path for reading the final.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three-layer separation: style learning (style_corpus), appeals-committee decisions
(internal_committee), and court rulings (external_upload).
- SCHEMA_V10: chair_name + district columns on case_law and cases, partial indexes
- create_internal_committee_decision() DB upsert function
- search_precedent_library_semantic() now accepts source_kind/district/chair_name params
- search_precedent_library_hybrid() passes through new params
- services/internal_decisions.py: ingest_internal_decision, migrate_from_style_corpus,
migrate_from_external_corpus (identifies rows via source_type='appeals_committee')
- search_internal_decisions() MCP tool (server.py + tools/search.py)
- internal_decision_migrate() MCP admin tool
- Web endpoints: POST /api/internal-decisions/upload, POST /api/internal-decisions/migrate,
GET /api/internal-decisions
- ingest_final_version auto-ingests finalized decisions into internal corpus
- SKILL.md updated: agents now search internal + external in parallel, present separately
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
block-he (פתיחה ניטרלית) was previously emitted only in final decisions.
For interim drafts shown to the chair before ruling, including a neutral
opening helps the chair confirm framing before approving downstream blocks.
Skipped if empty, so legacy cases without block-he are unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`rerank.maybe_rerank` calls `base_search(limit=…, **base_kwargs)` on both
the rerank-on and rerank-off paths. Commit 242f668 moved the closure into
hybrid_search.py and renamed its parameter to `limit_inner`, so every call
to `/api/precedent-library/search` raised TypeError 500 regardless of the
VOYAGE_RERANK_ENABLED flag. Sibling `search_documents_hybrid` was unaffected
because it uses `lambda **kw:` which absorbs the kwarg.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Observed 2026-05-03: a `precedent_process_pending(halacha)` run that
chained two precedents (1110/20 → 317/10) succeeded for the first
(9 halachot, 129 chunks) and produced status=`no_halachot` for the
second despite it being a 47KB Supreme Court ruling with rich legal
analysis. A manual single-precedent re-run on 317/10 immediately
extracted 53 halachot. Diagnosis: every chunk's claude_session call
in the back-to-back run silently failed (likely Anthropic rate-limit
storm after the 1110/20 token burn), and the empty list was reported
as "Claude looked and found nothing" — same code path as a real
0-halacha ruling. The user couldn't tell the difference.
Three changes:
1. Surface chunk-level failures (halacha_extractor.py)
`_extract_chunk` now returns `(halachot, succeeded)` so the caller
can count how many chunks crashed. `extract()` uses this to
distinguish:
- `no_halachot` — chunks ran cleanly, Claude found nothing
- `extraction_failed` — ≥50% of chunks crashed AND zero halachot
came back (rate limit, subprocess crash, etc.)
When `extraction_failed`, DB status is left as 'processing' so the
request stays in the queue for the caller to retry — instead of
the old behaviour where it got marked 'completed' and silently
dropped from the queue.
2. Inter-precedent cooldown (precedent_library.py)
`process_pending_extractions` now sleeps 30s between precedents.
Anthropic rate-limits per-org, and back-to-back large rulings
(~4M tokens for 1110/20, immediately followed by another 2-3M)
was the empirical trigger. 30s gives the per-minute counter time
to drain.
3. Auto-retry on extraction_failed (precedent_library.py)
When a precedent comes back as `extraction_failed`, retry once
after a 60s cooldown before giving up. Rate-limit storms are
transient — the manual re-run of 317/10 minutes later succeeded
with 53 halachot and zero chunk failures, confirming a single
retry is sufficient. Only retries `extraction_failed`; never
`no_halachot` (Claude looked and there genuinely is no holding).
The DB status now ends up as 'failed' only after retries are
exhausted, matching the UI's terminal-failure chip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When Paperclip wakes the CEO and the model issues an mcp__legal-ai__*
call within ~10s of session init, Claude Code sometimes returns
"No such tool available" because the legal-ai MCP server hasn't
finished bringing up its tool catalog yet. Observed twice today on
CMPA precedent-extraction wakeups (sessions 9989fbaf and a9c61801);
the agent fell back to bash + .venv/bin/python and finished the work,
but the race needed fixing on the server side.
Three changes that close the window:
1. Lazy schema init (services/db.py + server.py)
`init_schema()` was awaited inside the FastMCP lifespan, blocking
the `initialize`/`tools/list` handshake until ~10 CREATE TABLE IF
NOT EXISTS statements ran. Under contention (two CEOs waking at
once for different companies) this stretched. Now the lifespan
returns immediately and `get_pool()` runs the schema migrations
exactly once on first DB access, guarded by an asyncio.Lock.
tools/list is answered in milliseconds regardless of DB state.
2. Lazy heavy imports
- services/embeddings.py: voyageai (~450ms) loaded only inside
_get_client()
- services/extractor.py: google.cloud.vision (~550ms) loaded only
inside _get_vision_client() and _ocr_with_google_vision()
These two were being imported at module top from
legal_mcp.tools.documents -> services.processor -> services.{
extractor,embeddings}, so the FastMCP server couldn't even start
responding until both finished. Cold start dropped from 2.7s to
1.17s end-to-end (init + tools/list response).
3. Agent-side warmup + retry guidance (.claude/agents/legal-ceo.md)
Even with a fast server, the model can still race on the very
first call. The precedent-extraction section now tells the CEO
to call workflow_status as a warmup probe and to retry after a
short sleep if it sees "No such tool available", before falling
back to the python bypass.
Also expanded the precedent-tool whitelists on the sub-agents that
delegate halacha/library work (commits 4a9a6b7 + 7ee90dc added the
tools to the MCP server but only the CEO got them in its allowed
list). Added to: legal-researcher (full extraction set), legal-analyst
(library_get/list + halacha review), legal-writer (library lookups +
halacha_review), legal-qa (library_get + halacha_review), and the two
that the CEO was already missing (halacha_review, halachot_pending).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The legacy chunker did not track which PDF page each chunk came from.
Stored chunks had page_number=NULL, which blocked the multimodal
hybrid retriever's text+image boost — it joins (chunk, image) on
(document_id, page_number) and the join could never fire.
This change:
- extractor.extract_text now returns (text, page_count, page_offsets);
page_offsets[i] is the start char offset of page (i+1) in the joined
text. None for non-PDFs.
- chunker.chunk_document accepts an optional page_offsets and tags
each chunk with the page that contains its first character (uses
the existing chunker logic; pages assigned post-hoc by content
search to keep the diff minimal).
- processor.process_document and precedent_library.ingest_precedent
forward page_offsets through the chunker. New uploads now carry
accurate page_number on every chunk.
- Other extract_text callers (tools/documents, tools/workflow,
web/app.py) updated to unpack the third element (ignored).
- scripts/backfill_chunk_pages.py: per-case retrofit. Re-extracts each
PDF (re-OCRs via Google Vision if needed, ~$0.0015/page), computes
page_offsets, and updates page_number on every chunk by content
search. Idempotent; --force re-runs on already-tagged docs.
Forward-only would leave the 419 image embeddings backfilled on
cases 8174-24 + 8137-24 unable to boost their corresponding text
chunks. The retrofit script closes that gap (cost ~$0.60).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cosine scores in voyage-3 (~0.4-0.5) and voyage-multimodal-3
(~0.2-0.25) live on different scales. The previous weighted-sum
merge let text always dominate — verified empirically: 0 image-only
hits across 7 queries on case 8174-24, image side contributed nothing.
RRF combines by *rank* in each list rather than raw score, robust
to scale differences. Per-item score:
rrf_score = text_weight / (k + text_rank)
+ image_weight / (k + image_rank)
A row that appears in both lists (joined on (id_field, page_number))
gets both terms — surfaced as match_type='text+image'.
After fix on 8174-24 (146 image rows): 2 image-only hits land in
top-5 across all 7 test queries, surfacing actual table/diagram/
signature pages (p12, p13 of שומת המשיבה for 'טבלת השוואת ערכי שומה',
p25 of שומת השגה for 'תרשים גוש וחלקה', etc).
On 8137-24 (273 image rows): 'חישוב היוון של דמי החכירה' goes from
0 baseline results → 5 hybrid results (3 text + 2 image), opening
recall on scanned content the OCR layer misses.
Default MULTIMODAL_TEXT_WEIGHT 0.65 → 0.5 (vanilla RRF) since the
prior 0.65 was tuned for raw cosine scales that no longer apply.
New env knob MULTIMODAL_RRF_K (default 60, standard literature).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stage C: per-page image embeddings via voyage-multimodal-3 + hybrid
text+image search. Off by default; enable with MULTIMODAL_ENABLED=true.
- Schema V9: document_image_embeddings + precedent_image_embeddings
(vector(1024), page_number, image_thumbnail_path)
- extractor.render_pages_for_multimodal renders PDF pages at
MULTIMODAL_DPI (144) for embedding + JPEG thumbnails at
MULTIMODAL_THUMB_DPI (96) for UI preview, in one pass
- embeddings.embed_images calls voyage-multimodal-3 in 50-page batches
- services/hybrid_search.py orchestrator: rerank applied to text side
first (rerank-2 is text-only); image side cosine; weighted merge
with text_weight 0.65 (env-tunable); image-only pages surface as
match_type='image' so dense scanned content still appears
- processor.process_document and precedent_library.ingest_precedent
gated by flag — non-fatal on multimodal failure
- scripts/multimodal_backfill.py — idempotent per-case CLI to embed
existing documents without re-extracting text
Validated locally on a 5-page response brief: render 0.31s, embed 8.32s,
hybrid merge surfaces image rows correctly. Production rollout starts
with flag=false (no behavior change), then per-case A/B.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PATCH /api/halachot/{id} was returning 500 because the row included
``embedding`` as a numpy.ndarray of np.float32, which FastAPI's
jsonable_encoder cannot serialize (vars() and dict() both fail on it).
The bug had been latent — it triggered for the first time today after
the auto-approve batch left only low-confidence halachot for the chair
to review manually, and her first PATCH hit the unserializable response.
Replace ``RETURNING *`` with an explicit column list (everything except
``embedding``). Callers that need the embedding can re-fetch via
``get_halacha`` — but no current caller does.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
