feat: Stage C — RAG advanced (#33, #47, #48, #49, #50, #51)
All checks were successful
Build & Deploy / build-and-deploy (push) Successful in 1m35s
All checks were successful
Build & Deploy / build-and-deploy (push) Successful in 1m35s
Six independent sub-tasks dispatched in parallel; aggregated here. ## #33 — Hide case_name column library-list-panel.tsx: `<TableHead>` + `<TableCell>` for "שם" get `className="hidden"` in both Court and Committee row variants. DB column preserved for future use. ## #47 — Audit script periodic New scripts/audit_corpus_integrity.py — 3 SQL checks (external+ערר prefix, internal missing chair/district, cases.practice_area enum) + CEO wakeup on violations + cron `0 7 * * *`. First run: 0 issues. ## #48 — Parent-doc retrieval (gated, default off) Schema V17: precedent_chunks.parent_chunk_id + chunk_role ('child'|'parent'). New chunker.chunk_document_hierarchical() — section-aware parents (~1500 tokens) containing ~5 overlapping children (~300 tokens each). New db.store_precedent_chunks_hierarchical two-pass writer. Search SQL (semantic + lexical) LEFT-JOIN parent and swap content + dedupe by parent_chunk_id when flag on. Toggle: PARENT_DOC_RETRIEVAL_ENABLED + PARENT_DOC_{CHILD,PARENT}_SIZE_TOKENS. Backfill ~3min and ~$0.20 — deferred to follow-up. ## #49 — Multimodal backfill New scripts/backfill_multimodal_precedents.py with token-matching case_number ↔ source files (PDF + DOCX via PyMuPDF). Ran in container: 26 precedents embedded, 503 pages, $0.21, 0 errors. precedent_image_embeddings grew 3 → 29 rows. 44 remaining are style_corpus-migrated rows (no source file on disk) — will catch up when re-uploaded. ## #50 — Closed-loop feedback + nDCG Schema V18: search_logs + search_relevance_feedback. New telemetry.py with fire-and-forget log_search_bg (p50 = 0.002ms — zero overhead) + auto-infer_relevance_from_citations (reads case drafts → marks score=3 when cited precedent appears in past search top-K). Hooks added to 5 search paths. scripts/compute_ndcg.py for aggregation. Two admin API endpoints (GET /api/admin/rag-metrics + POST .../infer). Dashboard UI deferred — API is enough for now. ## #51 — Halacha quality monitoring New scripts/monitor_halacha_quality.py — baseline avg confidence (trusted=0.849, all=0.833, pending=0.694) with rolling window drift detection. Default 5% threshold. Exits non-zero on alert for cron integration. Recommended: `0 8 * * 1` weekly Mon 8am. ## Bonus: 230 unlinked citations → missing_precedents Bulk-imported 230 distinct unlinked citations from precedent_internal_citations to missing_precedents.status='open', party='committee', with notes listing source citers. Top candidate: ע"א 3213/97 (cited 5x). Total open missing_precedents now 237. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,4 +1,14 @@
|
||||
"""Legal document chunker - splits text into sections and chunks for RAG."""
|
||||
"""Legal document chunker - splits text into sections and chunks for RAG.
|
||||
|
||||
The default :func:`chunk_document` emits a single tier of overlapping
|
||||
chunks (legacy single-tier indexing). :func:`chunk_document_hierarchical`
|
||||
emits two tiers — small "child" chunks for retrieval matching, plus
|
||||
larger "parent" chunks that supply broader context to the LLM (parent-
|
||||
doc retrieval, TaskMaster #48). The hierarchical variant lives
|
||||
alongside the legacy one so callers can opt in via
|
||||
``config.PARENT_DOC_RETRIEVAL_ENABLED`` without breaking existing
|
||||
single-tier code paths.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -162,3 +172,152 @@ def _split_section(text: str, chunk_size: int, overlap: int) -> list[str]:
|
||||
def _estimate_tokens(text: str) -> int:
|
||||
"""Rough token estimate for Hebrew text (~1.5 chars per token)."""
|
||||
return max(1, len(text) // 2)
|
||||
|
||||
|
||||
# ── Parent-doc retrieval (TaskMaster #48) ────────────────────────────
|
||||
# Hierarchical chunker — emits a list of (child, parent) pairs:
|
||||
# * each "child" carries the smaller text used for embedding/search
|
||||
# * each "parent" is shared by ~5 consecutive children (1500/300)
|
||||
# The list is FLAT — both parents and children live in the same return
|
||||
# list, distinguished by ``role``. A child's ``parent_local_id`` points
|
||||
# back to its parent's ``local_id``, so the ingest pipeline can resolve
|
||||
# the FK after the parent row is INSERTed and its DB UUID is known.
|
||||
#
|
||||
# Parents are built FIRST (one window of ``parent_size`` tokens per
|
||||
# section, sliding by the parent window — no overlap between parents),
|
||||
# then each parent is sub-divided into overlapping children. This keeps
|
||||
# the parent boundary aligned with semantic sections (so a "discussion"
|
||||
# parent doesn't contain stray "ruling" prose) while still allowing
|
||||
# child overlap for recall.
|
||||
|
||||
|
||||
@dataclass
|
||||
class HierarchicalChunk:
|
||||
"""One chunk in the two-tier hierarchy.
|
||||
|
||||
Both children and parents share this shape; ``role`` distinguishes
|
||||
them. Children get an embedding at ingest time; parents do not —
|
||||
they exist only to carry context back to the LLM at retrieval time.
|
||||
|
||||
``local_id`` is a stable in-batch identifier (sequential int) used
|
||||
only by the ingest pipeline to wire children to their parent's DB
|
||||
UUID after the parent INSERT returns. It is NOT persisted.
|
||||
"""
|
||||
|
||||
content: str
|
||||
role: str # 'child' | 'parent'
|
||||
section_type: str = "other"
|
||||
page_number: int | None = None
|
||||
chunk_index: int = 0
|
||||
local_id: int = -1
|
||||
parent_local_id: int | None = None
|
||||
|
||||
|
||||
def chunk_document_hierarchical(
|
||||
text: str,
|
||||
child_size: int = config.PARENT_DOC_CHILD_SIZE_TOKENS,
|
||||
parent_size: int = config.PARENT_DOC_PARENT_SIZE_TOKENS,
|
||||
overlap: int = config.PARENT_DOC_CHILD_OVERLAP_TOKENS,
|
||||
page_offsets: list[int] | None = None,
|
||||
) -> list[HierarchicalChunk]:
|
||||
"""Split a document into a two-tier (child, parent) hierarchy.
|
||||
|
||||
Returns a flat list where each element is either a parent or a
|
||||
child. Children carry ``parent_local_id`` pointing back to their
|
||||
parent's ``local_id``. Caller (ingest pipeline) must insert parents
|
||||
first, capture their DB UUIDs by ``local_id``, then insert children
|
||||
with the resolved UUID in ``parent_chunk_id``.
|
||||
|
||||
Args:
|
||||
text: full document text.
|
||||
child_size: child chunk size in tokens (≈ 300 by default).
|
||||
parent_size: parent chunk size in tokens (≈ 1500 by default).
|
||||
Parents contain ``parent_size // child_size`` children on
|
||||
average.
|
||||
overlap: child-to-child overlap inside a parent (≈ 50 tokens).
|
||||
Parents themselves do not overlap each other.
|
||||
page_offsets: PDF page offsets for tagging chunks with page #.
|
||||
|
||||
Notes:
|
||||
* Parents respect section boundaries (header detection from
|
||||
:data:`SECTION_PATTERNS`). A "facts" parent will not include
|
||||
"ruling" text.
|
||||
* Empty text returns an empty list.
|
||||
* Both child and parent rows are tagged with the page of their
|
||||
first character.
|
||||
"""
|
||||
if not text.strip():
|
||||
return []
|
||||
if child_size <= 0 or parent_size <= 0:
|
||||
raise ValueError("child_size and parent_size must be positive")
|
||||
if child_size > parent_size:
|
||||
raise ValueError("child_size must be <= parent_size")
|
||||
|
||||
sections = _split_into_sections(text)
|
||||
out: list[HierarchicalChunk] = []
|
||||
parent_idx = 0 # global parent ordinal (chunk_index for parents)
|
||||
child_idx = 0 # global child ordinal (chunk_index for children)
|
||||
local_id = 0 # sequential id within this document
|
||||
|
||||
for section_type, section_text in sections:
|
||||
# Step 1: split section into parent-sized windows (no overlap).
|
||||
parent_texts = _split_section(section_text, parent_size, overlap=0)
|
||||
for parent_text in parent_texts:
|
||||
parent_local = local_id
|
||||
local_id += 1
|
||||
parent_chunk = HierarchicalChunk(
|
||||
content=parent_text,
|
||||
role="parent",
|
||||
section_type=section_type,
|
||||
chunk_index=parent_idx,
|
||||
local_id=parent_local,
|
||||
parent_local_id=None,
|
||||
)
|
||||
out.append(parent_chunk)
|
||||
parent_idx += 1
|
||||
|
||||
# Step 2: sub-divide this parent into overlapping children.
|
||||
child_texts = _split_section(parent_text, child_size, overlap)
|
||||
for ch_text in child_texts:
|
||||
ch = HierarchicalChunk(
|
||||
content=ch_text,
|
||||
role="child",
|
||||
section_type=section_type,
|
||||
chunk_index=child_idx,
|
||||
local_id=local_id,
|
||||
parent_local_id=parent_local,
|
||||
)
|
||||
out.append(ch)
|
||||
local_id += 1
|
||||
child_idx += 1
|
||||
|
||||
if page_offsets:
|
||||
_assign_pages_hierarchical(out, text, page_offsets)
|
||||
return out
|
||||
|
||||
|
||||
def _assign_pages_hierarchical(
|
||||
chunks: list[HierarchicalChunk],
|
||||
text: str,
|
||||
page_offsets: list[int],
|
||||
) -> None:
|
||||
"""Page-tag both children and parents.
|
||||
|
||||
Same forward-scan strategy as :func:`_assign_pages` but works on
|
||||
the hierarchical list. Parents may span pages; we tag them with
|
||||
the page of their first character (matches how the multimodal
|
||||
retriever joins on page numbers).
|
||||
"""
|
||||
from legal_mcp.services.extractor import page_at_offset
|
||||
pos = 0
|
||||
for c in chunks:
|
||||
idx = text.find(c.content, pos)
|
||||
if idx < 0:
|
||||
idx = text.find(c.content)
|
||||
if idx < 0:
|
||||
continue
|
||||
c.page_number = page_at_offset(idx, page_offsets)
|
||||
# Advance past halfway — children share text with their parent
|
||||
# and with each other (overlap), so a small forward step lets
|
||||
# the next find() still pick up the right occurrence.
|
||||
pos = idx + max(1, len(c.content) // 4)
|
||||
|
||||
Reference in New Issue
Block a user