The legacy chunker did not track which PDF page each chunk came from.
Stored chunks had page_number=NULL, which blocked the multimodal
hybrid retriever's text+image boost — it joins (chunk, image) on
(document_id, page_number) and the join could never fire.
This change:
- extractor.extract_text now returns (text, page_count, page_offsets);
page_offsets[i] is the start char offset of page (i+1) in the joined
text. None for non-PDFs.
- chunker.chunk_document accepts an optional page_offsets and tags
each chunk with the page that contains its first character (uses
the existing chunker logic; pages assigned post-hoc by content
search to keep the diff minimal).
- processor.process_document and precedent_library.ingest_precedent
forward page_offsets through the chunker. New uploads now carry
accurate page_number on every chunk.
- Other extract_text callers (tools/documents, tools/workflow,
web/app.py) updated to unpack the third element (ignored).
- scripts/backfill_chunk_pages.py: per-case retrofit. Re-extracts each
PDF (re-OCRs via Google Vision if needed, ~$0.0015/page), computes
page_offsets, and updates page_number on every chunk by content
search. Idempotent; --force re-runs on already-tagged docs.
Forward-only would leave the 419 image embeddings backfilled on
cases 8174-24 + 8137-24 unable to boost their corresponding text
chunks. The retrofit script closes that gap (cost ~$0.60).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cases failed to push silently after the Gitea token in Infisical was
rotated: the embedded credential in each case repo's origin URL was
the old token, the rotation never propagated, and capture_output=True
hid the auth failure as a logger.warning. Three cases (1033-25,
1130-25, 1194-25) accumulated unpushed commits over weeks before
this was noticed.
Fixes the root cause in two places: web/gitea_client.py for uploads
through the FastAPI endpoint, and mcp-server/services/git_sync.py
for case_update / document_upload through MCP tools (which previously
committed but never pushed at all).
The new commit_and_push helper:
- re-injects the current GITEA_ACCESS_TOKEN into the existing origin
URL on every call, so pushes survive token rotation
- logs push failures at WARNING with the actual stderr (the previous
code suppressed errors entirely)
- continues to push even when the commit was a no-op, in case earlier
commits are still unpushed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Until now changing a document's doc_type required a manual SQL update.
Adds an inline editor on the document badge so the chair can retag
without leaving the case page, and threads an appraiser_side tag
(committee / appellant / deciding) through the appraisal pipeline so
betterment-levy cases — which usually have 2-3 appraisers — render
conflicts with the deciding appraiser's view marked as governing.
Backend
- New appraiser_facts.appraiser_side column (V5.1) populated from
documents.metadata.appraiser_side at extraction time.
- extract_appraiser_facts now returns status='sides_missing' with the
list of untagged appraisals instead of running with empty side
labels — chair must tag every appraisal first via the UI.
- Conflict detection orders entries committee → appellant → deciding so
the deciding appraiser appears last; block-tet's prompt instructs the
writer to phrase the deciding appraiser's view as the governing
factual finding ("ואולם, השמאי המכריע קבע...").
- New PATCH /api/cases/{n}/documents/{doc_id} (Pydantic model with
whitelist validation) and matching document_update MCP tool. Both
merge appraiser_side into metadata JSONB instead of touching the
schema.
UI
- New shared doc-types module exports the canonical 11 doc_type
options plus the 3 appraiser-side options; both upload-sheet and
the document badge now read from it instead of duplicating Hebrew
labels.
- New DocumentTypeEditor renders a Popover off the doc-type Badge
with two Selects. The save button stays disabled while doc_type is
appraisal but no side has been picked, mirroring the backend
enforcement so the user finds out before triggering extraction.
- usePatchDocument React-Query mutation invalidates the case detail
on success so the badge updates without a manual refresh.
- Add strip_nevo_preamble() to extractor.py — auto-removes Nevo database
headers (bibliography, legislation, mini-ratio) during training upload
- Add appeal_subtype column to style_patterns table — patterns are now
stored per subtype instead of globally mixed
- Update clear_style_patterns() to support subtype-scoped deletion
- Pass appeal_subtype through analyze_corpus → store → upsert pipeline
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Support ingestion of betterment levy (היטל השבחה) decisions into a
separate training corpus (CMPA). Key changes:
- Add .doc file extraction via LibreOffice conversion in extractor
- Add practice_area/appeal_subtype columns to style_corpus table
- Route training files to cmp/ or cmpa/ subdirs based on appeal subtype
- Fix derive_subtype to handle ARAR-YY-NNNN format (was matching year digit)
- Expose practice_area/appeal_subtype params in MCP upload_training tool
- Add appeal_subtype filter to analyze_style for per-type style analysis
- Update betterment levy methodology in lessons.py: checklist (from generic
to corpus-based), opening/closing strategies, and discussion rules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Install git in Docker image and wrap all subprocess git calls in
try/except so a missing or failing git binary never kills an upload
that already succeeded.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds two orthogonal columns — practice_area (top-level legal domain:
appeals_committee / national_insurance / labor_law) and appeal_subtype
(building_permit / betterment_levy / compensation_197) — denormalized
into cases, documents, document_chunks, decisions, and style_corpus so
vector searches can filter without JOINs.
Why: the system handles two unrelated sub-domains under the same
appeals committee (1xxx building permits and 8xxx/9xxx betterment/197),
with different rules and writing style. Without a separation axis,
search_similar() and the block-writer's precedent lookup were free to
surface betterment-levy paragraphs while drafting a building-permit
decision — a real risk of cross-domain contamination. The same axis
also lets future domains (national insurance, labor law) coexist
without separate schemas.
Schema (V4 migration in db.py):
- ALTER ... ADD COLUMN IF NOT EXISTS on all five tables + composite
indexes (practice_area first).
- Idempotent backfill: case_number ~ '^1' → building_permit, '^8' →
betterment_levy, '^9' → compensation_197; propagated to documents,
chunks, and decisions via case_id; training-corpus rows (case_id NULL)
default to appeals_committee.
Code:
- New services/practice_area.py with derive_subtype, validate, and
is_override + enum constants.
- db.create_case / create_document / store_chunks / create_decision
inherit practice_area from the parent case (or take an explicit
override for the case_id=None training corpus).
- db.search_similar and search_similar_paragraphs accept practice_area
+ appeal_subtype filters using the denormalized columns.
- tools/search.py auto-resolves the filter from case_number when given.
- block_writer._build_precedents_context now passes the active case's
practice_area to search_similar_paragraphs — closes the contamination
hole for the discussion-block precedent fetch.
- tools/cases.case_create auto-derives subtype from case_number; an
explicit override that disagrees writes a case_subtype_override entry
to audit_log so we can spot bad classifications later.
- tools/documents.document_upload_training tags new training material
with practice_area + subtype end-to-end (corpus, document, chunks).
UI (web/static/index.html + web/app.py):
- New-case wizard gets a practice_area dropdown (others disabled until
national_insurance / labor_law arrive) and an appeal_subtype dropdown
with JS auto-fill from the case-number prefix; manual edits stick.
- Case header shows a blue badge with practice_area · subtype.
- CaseCreateRequest plumbs both fields through to cases_tools.case_create.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove cases/new|in-progress|completed subdivision (status managed in DB)
- Rename documents/original → documents/originals (consistent plural)
- Move exports from global data/exports/ into cases/{num}/exports/
- Add documents/research/ for case law and analysis files
- Update all agents, scripts, config, web API endpoints, and DB paths
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace single CASES_DIR with find_case_dir() that searches across
all status directories. New cases created in cases/new/{number}/.
Config: CASES_BASE, CASES_NEW, CASES_IN_PROGRESS, CASES_COMPLETED
Docker: added -v /home/chaim/legal-ai/cases:/cases volume mount
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ezer Mishpati - AI legal decision drafting system with:
- MCP server (FastMCP) with document processing pipeline
- Web upload interface (FastAPI) for file upload and classification
- pgvector-based semantic search
- Hebrew legal document chunking and embedding
