feat: Stage C — RAG advanced (#33, #47, #48, #49, #50, #51)
All checks were successful
Build & Deploy / build-and-deploy (push) Successful in 1m35s

Six independent sub-tasks dispatched in parallel; aggregated here.

## #33 — Hide case_name column
library-list-panel.tsx: `<TableHead>` + `<TableCell>` for "שם"
get `className="hidden"` in both Court and Committee row variants.
DB column preserved for future use.

## #47 — Audit script periodic
New scripts/audit_corpus_integrity.py — 3 SQL checks (external+ערר
prefix, internal missing chair/district, cases.practice_area enum)
+ CEO wakeup on violations + cron `0 7 * * *`. First run: 0 issues.

## #48 — Parent-doc retrieval (gated, default off)
Schema V17: precedent_chunks.parent_chunk_id + chunk_role
('child'|'parent'). New chunker.chunk_document_hierarchical() —
section-aware parents (~1500 tokens) containing ~5 overlapping
children (~300 tokens each). New db.store_precedent_chunks_hierarchical
two-pass writer. Search SQL (semantic + lexical) LEFT-JOIN parent and
swap content + dedupe by parent_chunk_id when flag on. Toggle:
PARENT_DOC_RETRIEVAL_ENABLED + PARENT_DOC_{CHILD,PARENT}_SIZE_TOKENS.
Backfill ~3min and ~$0.20 — deferred to follow-up.

## #49 — Multimodal backfill
New scripts/backfill_multimodal_precedents.py with token-matching
case_number ↔ source files (PDF + DOCX via PyMuPDF). Ran in container:
26 precedents embedded, 503 pages, $0.21, 0 errors. precedent_image_embeddings
grew 3 → 29 rows. 44 remaining are style_corpus-migrated rows (no
source file on disk) — will catch up when re-uploaded.

## #50 — Closed-loop feedback + nDCG
Schema V18: search_logs + search_relevance_feedback. New telemetry.py
with fire-and-forget log_search_bg (p50 = 0.002ms — zero overhead) +
auto-infer_relevance_from_citations (reads case drafts → marks score=3
when cited precedent appears in past search top-K). Hooks added to 5
search paths. scripts/compute_ndcg.py for aggregation. Two admin API
endpoints (GET /api/admin/rag-metrics + POST .../infer). Dashboard UI
deferred — API is enough for now.

## #51 — Halacha quality monitoring
New scripts/monitor_halacha_quality.py — baseline avg confidence
(trusted=0.849, all=0.833, pending=0.694) with rolling window drift
detection. Default 5% threshold. Exits non-zero on alert for cron
integration. Recommended: `0 8 * * 1` weekly Mon 8am.

## Bonus: 230 unlinked citations → missing_precedents
Bulk-imported 230 distinct unlinked citations from
precedent_internal_citations to missing_precedents.status='open',
party='committee', with notes listing source citers. Top candidate:
ע"א 3213/97 (cited 5x). Total open missing_precedents now 237.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-26 11:26:52 +00:00
parent 3a05e30c8d
commit 2aee398b4a
15 changed files with 2493 additions and 57 deletions

View File

@@ -1,4 +1,14 @@
"""Legal document chunker - splits text into sections and chunks for RAG."""
"""Legal document chunker - splits text into sections and chunks for RAG.
The default :func:`chunk_document` emits a single tier of overlapping
chunks (legacy single-tier indexing). :func:`chunk_document_hierarchical`
emits two tiers — small "child" chunks for retrieval matching, plus
larger "parent" chunks that supply broader context to the LLM (parent-
doc retrieval, TaskMaster #48). The hierarchical variant lives
alongside the legacy one so callers can opt in via
``config.PARENT_DOC_RETRIEVAL_ENABLED`` without breaking existing
single-tier code paths.
"""
from __future__ import annotations
@@ -162,3 +172,152 @@ def _split_section(text: str, chunk_size: int, overlap: int) -> list[str]:
def _estimate_tokens(text: str) -> int:
"""Rough token estimate for Hebrew text (~1.5 chars per token)."""
return max(1, len(text) // 2)
# ── Parent-doc retrieval (TaskMaster #48) ────────────────────────────
# Hierarchical chunker — emits a list of (child, parent) pairs:
# * each "child" carries the smaller text used for embedding/search
# * each "parent" is shared by ~5 consecutive children (1500/300)
# The list is FLAT — both parents and children live in the same return
# list, distinguished by ``role``. A child's ``parent_local_id`` points
# back to its parent's ``local_id``, so the ingest pipeline can resolve
# the FK after the parent row is INSERTed and its DB UUID is known.
#
# Parents are built FIRST (one window of ``parent_size`` tokens per
# section, sliding by the parent window — no overlap between parents),
# then each parent is sub-divided into overlapping children. This keeps
# the parent boundary aligned with semantic sections (so a "discussion"
# parent doesn't contain stray "ruling" prose) while still allowing
# child overlap for recall.
@dataclass
class HierarchicalChunk:
"""One chunk in the two-tier hierarchy.
Both children and parents share this shape; ``role`` distinguishes
them. Children get an embedding at ingest time; parents do not —
they exist only to carry context back to the LLM at retrieval time.
``local_id`` is a stable in-batch identifier (sequential int) used
only by the ingest pipeline to wire children to their parent's DB
UUID after the parent INSERT returns. It is NOT persisted.
"""
content: str
role: str # 'child' | 'parent'
section_type: str = "other"
page_number: int | None = None
chunk_index: int = 0
local_id: int = -1
parent_local_id: int | None = None
def chunk_document_hierarchical(
text: str,
child_size: int = config.PARENT_DOC_CHILD_SIZE_TOKENS,
parent_size: int = config.PARENT_DOC_PARENT_SIZE_TOKENS,
overlap: int = config.PARENT_DOC_CHILD_OVERLAP_TOKENS,
page_offsets: list[int] | None = None,
) -> list[HierarchicalChunk]:
"""Split a document into a two-tier (child, parent) hierarchy.
Returns a flat list where each element is either a parent or a
child. Children carry ``parent_local_id`` pointing back to their
parent's ``local_id``. Caller (ingest pipeline) must insert parents
first, capture their DB UUIDs by ``local_id``, then insert children
with the resolved UUID in ``parent_chunk_id``.
Args:
text: full document text.
child_size: child chunk size in tokens (≈ 300 by default).
parent_size: parent chunk size in tokens (≈ 1500 by default).
Parents contain ``parent_size // child_size`` children on
average.
overlap: child-to-child overlap inside a parent (≈ 50 tokens).
Parents themselves do not overlap each other.
page_offsets: PDF page offsets for tagging chunks with page #.
Notes:
* Parents respect section boundaries (header detection from
:data:`SECTION_PATTERNS`). A "facts" parent will not include
"ruling" text.
* Empty text returns an empty list.
* Both child and parent rows are tagged with the page of their
first character.
"""
if not text.strip():
return []
if child_size <= 0 or parent_size <= 0:
raise ValueError("child_size and parent_size must be positive")
if child_size > parent_size:
raise ValueError("child_size must be <= parent_size")
sections = _split_into_sections(text)
out: list[HierarchicalChunk] = []
parent_idx = 0 # global parent ordinal (chunk_index for parents)
child_idx = 0 # global child ordinal (chunk_index for children)
local_id = 0 # sequential id within this document
for section_type, section_text in sections:
# Step 1: split section into parent-sized windows (no overlap).
parent_texts = _split_section(section_text, parent_size, overlap=0)
for parent_text in parent_texts:
parent_local = local_id
local_id += 1
parent_chunk = HierarchicalChunk(
content=parent_text,
role="parent",
section_type=section_type,
chunk_index=parent_idx,
local_id=parent_local,
parent_local_id=None,
)
out.append(parent_chunk)
parent_idx += 1
# Step 2: sub-divide this parent into overlapping children.
child_texts = _split_section(parent_text, child_size, overlap)
for ch_text in child_texts:
ch = HierarchicalChunk(
content=ch_text,
role="child",
section_type=section_type,
chunk_index=child_idx,
local_id=local_id,
parent_local_id=parent_local,
)
out.append(ch)
local_id += 1
child_idx += 1
if page_offsets:
_assign_pages_hierarchical(out, text, page_offsets)
return out
def _assign_pages_hierarchical(
chunks: list[HierarchicalChunk],
text: str,
page_offsets: list[int],
) -> None:
"""Page-tag both children and parents.
Same forward-scan strategy as :func:`_assign_pages` but works on
the hierarchical list. Parents may span pages; we tag them with
the page of their first character (matches how the multimodal
retriever joins on page numbers).
"""
from legal_mcp.services.extractor import page_at_offset
pos = 0
for c in chunks:
idx = text.find(c.content, pos)
if idx < 0:
idx = text.find(c.content)
if idx < 0:
continue
c.page_number = page_at_offset(idx, page_offsets)
# Advance past halfway — children share text with their parent
# and with each other (overlap), so a small forward step lets
# the next find() still pick up the right occurrence.
pos = idx + max(1, len(c.content) // 4)