feat(retrieval): track page_number on text chunks for multimodal hybrid boost
All checks were successful
Build & Deploy / build-and-deploy (push) Successful in 6m33s

The legacy chunker did not track which PDF page each chunk came from.
Stored chunks had page_number=NULL, which blocked the multimodal
hybrid retriever's text+image boost — it joins (chunk, image) on
(document_id, page_number) and the join could never fire.

This change:

- extractor.extract_text now returns (text, page_count, page_offsets);
  page_offsets[i] is the start char offset of page (i+1) in the joined
  text. None for non-PDFs.
- chunker.chunk_document accepts an optional page_offsets and tags
  each chunk with the page that contains its first character (uses
  the existing chunker logic; pages assigned post-hoc by content
  search to keep the diff minimal).
- processor.process_document and precedent_library.ingest_precedent
  forward page_offsets through the chunker. New uploads now carry
  accurate page_number on every chunk.
- Other extract_text callers (tools/documents, tools/workflow,
  web/app.py) updated to unpack the third element (ignored).
- scripts/backfill_chunk_pages.py: per-case retrofit. Re-extracts each
  PDF (re-OCRs via Google Vision if needed, ~$0.0015/page), computes
  page_offsets, and updates page_number on every chunk by content
  search. Idempotent; --force re-runs on already-tagged docs.

Forward-only would leave the 419 image embeddings backfilled on
cases 8174-24 + 8137-24 unable to boost their corresponding text
chunks. The retrofit script closes that gap (cost ~$0.60).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-03 19:49:41 +00:00
parent 5724ed8e5b
commit 81ccf3a888
9 changed files with 301 additions and 18 deletions

View File

@@ -144,7 +144,7 @@ async def document_upload_training(
shutil.copy2(str(source), str(dest))
# Extract text and strip Nevo preamble
text, page_count = await extractor.extract_text(str(dest))
text, page_count, _ = await extractor.extract_text(str(dest))
text = extractor.strip_nevo_preamble(text)
# Parse date

View File

@@ -308,7 +308,7 @@ async def ingest_final_version(
# Extract text from file if provided
if file_path and not final_text:
from legal_mcp.services import extractor
final_text, _ = await extractor.extract_text(file_path)
final_text, _, _ = await extractor.extract_text(file_path)
if not final_text:
return "לא סופק טקסט — יש לספק file_path או final_text."