fix(extraction): סינון cited_only מתור/מוני החילוץ (#140)
31 שורות case_law עם source_kind='cited_only' (ציטוט-בלבד, ללא full_text/chunks) נושאות halacha_extraction_status='pending' רק כברירת-מחדל ומזהמות את מונה ה-pending ובמתזמר/בדף-התפעול — אין להן מה לחלץ. תיקון (G1 — תיקון-במקור, G2 — מסנן יחיד משותף): - db.EXTRACTION_ELIGIBLE_PREDICATE — מקור-אמת יחיד ל"שורה ברת-חילוץ" (source_kind <> 'cited_only' AND יש precedent_chunks). מוחל ב-list_pending_extraction_requests; #139 יעשה בו שימוש-חוזר ל-reconcile (אותו כלל, לא כפול). - מוני-snapshot מסננים cited_only: halacha_drain_supervisor.db_snapshot, web/app.py meta+hal_ext (GROUP BY status). - reconcile_metadata_status.py מורחב לכסות גם את תור-ההלכות: cited_only→'skipped' (אותו terminal-state כמו צד-המטא, תור-תאום, G2). בוצע על ה-DB החי: 31 הועברו ל-'skipped' (metadata כבר היה מיושב — אידמפוטנטי). התפלגות-אחרי: halacha pending=9 (עבודה אמיתית), skipped=31, completed=309. בדיקות: test_extraction_queue_eligibility (predicate + list_pending מחיל אותו, שני ה-kinds). כל 345 בדיקות mcp עוברות. guards נקיים. Invariants: G1 (terminal-state אמיתי במקור), G2 (predicate יחיד, ללא תור מקביל), INV-DM1 (stub לא-searchable אינו מועמד-חילוץ), G12 (leak-guard נקי). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
"""Reconcile stale ``metadata_extraction_status='pending'`` rows (G1).
|
||||
"""Reconcile stale ``*_extraction_status='pending'`` rows (G1).
|
||||
|
||||
The column defaults to 'pending', but only ``source_kind='external_upload'``
|
||||
rows with extractable text genuinely need the Gemini metadata drain. Internal
|
||||
@@ -13,6 +13,10 @@ This settles each row to a truthful terminal state at the source:
|
||||
- external_upload w/ text but missing name/summary → stamp requested_at (real work → drain picks it up)
|
||||
- cited_only (no text) → 'skipped' (terminal; nothing to extract)
|
||||
|
||||
Covers BOTH extraction queues (#140): the cited_only→'skipped' settle is applied
|
||||
to ``halacha_extraction_status`` as well as ``metadata_extraction_status`` — same
|
||||
phantom-backlog fix on the twin queue, one reconcile script (G2).
|
||||
|
||||
Idempotent and re-runnable (a healthy DB reports all-zero). The companion source
|
||||
fix lives in db.create_internal_committee_decision (inserts 'completed' directly)
|
||||
so internal rows never re-enter this state.
|
||||
@@ -57,6 +61,15 @@ async def main() -> int:
|
||||
"WHERE source_kind = 'cited_only' "
|
||||
"AND metadata_extraction_status = 'pending'"
|
||||
)
|
||||
# Halacha side (#140): cited_only stubs inherit DEFAULT 'pending' but have no
|
||||
# text/chunks to extract holdings from — settle them to the same terminal
|
||||
# 'skipped' the metadata side uses, so they stop inflating the halacha
|
||||
# pending counter / supervisor snapshot. Same source-fix, one reconcile (G2).
|
||||
cited_hal = await pool.execute(
|
||||
"UPDATE case_law SET halacha_extraction_status = 'skipped' "
|
||||
"WHERE source_kind = 'cited_only' "
|
||||
"AND halacha_extraction_status = 'pending'"
|
||||
)
|
||||
|
||||
def n(tag: str) -> str:
|
||||
try:
|
||||
@@ -64,10 +77,11 @@ async def main() -> int:
|
||||
except (AttributeError, IndexError):
|
||||
return "?"
|
||||
|
||||
print(f"internal_committee → completed : {n(internal)}")
|
||||
print(f"external_upload → completed : {n(external_done)}")
|
||||
print(f"external_upload → requeued : {n(external_requeued)}")
|
||||
print(f"cited_only → skipped : {n(cited)}")
|
||||
print(f"internal_committee → completed : {n(internal)}")
|
||||
print(f"external_upload → completed : {n(external_done)}")
|
||||
print(f"external_upload → requeued : {n(external_requeued)}")
|
||||
print(f"cited_only metadata → skipped : {n(cited)}")
|
||||
print(f"cited_only halacha → skipped : {n(cited_hal)}")
|
||||
|
||||
rows = await pool.fetch(
|
||||
"SELECT coalesce(metadata_extraction_status,'NULL') s, count(*) n "
|
||||
@@ -76,6 +90,13 @@ async def main() -> int:
|
||||
print("\nresulting metadata_extraction_status distribution:")
|
||||
for r in rows:
|
||||
print(f" {r['s']:<12} {r['n']}")
|
||||
hal_rows = await pool.fetch(
|
||||
"SELECT coalesce(halacha_extraction_status,'NULL') s, count(*) n "
|
||||
"FROM case_law GROUP BY 1 ORDER BY 2 DESC"
|
||||
)
|
||||
print("\nresulting halacha_extraction_status distribution:")
|
||||
for r in hal_rows:
|
||||
print(f" {r['s']:<12} {r['n']}")
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user