Upgrade: speed up halacha+metadata extraction #433

New Issue

chaim · 2026-08-05T11:01:10Z

chaim commented

2026-08-05 11:01:10 +00:00

What & Why

Halacha extraction on long rulings is slow (5-15 min for typical court ruling, 30-50 min for a 207-chunk appeals committee decision). Root cause: each chunk spawns a separate claude -p subprocess (5-10 sec startup overhead each), Hebrew prompts on cold cache run 30-90 sec, and there's no prompt-cache sharing between chunks. Acceleration options to evaluate later when speed becomes a real blocker.

Options (each can be combined):

Concurrency 3 -> 6 in halacha_extractor.CHUNK_CONCURRENCY. ~2x faster wall-clock. Cost: 6x ~300MB RSS = 1.8GB peak — verify on Nautilus headroom.
Larger chunks 12K -> 18-25K chars (CHUNK_TARGET_CHARS in claims_extractor.py / halacha_extractor.py). Fewer waves. Risk: timeout on cold cache (currently 1800s ceiling), and may degrade extraction precision for very long sections.
Anthropic SDK direct with 5-min ephemeral prompt caching on the static instruction prefix (already wired the parameter as system= in claude_session.query). Estimated 5-10x faster because cache reads are ~10% of cold cost. Costs ~$0.30-2 per long ruling on Sonnet 4.6. Chair previously rejected this path for ALL traffic ('we work only with claude session'). Compromise: SDK only for the precedent-library corpus build (static, one-time), claude session for live decision drafting (interactive, frequent).
Two-tier prompt: a short 'classification' pass with claude -p deciding which chunks contain halachot, then deep extraction only on positive chunks. Could cut total LLM time by 40-60% on rulings with lots of factual chapters.
Already implemented (Apr 3, 2026): skip non-extractable sections — only run on chunks where section_type IN (legal_analysis, ruling, conclusion); fallback to all chunks when chunker labels nothing. So that win is already banked.

Re-evaluate when: a chair drops a 200K+ char ruling into the queue and the wait becomes painful, OR when the precedent-library has 50+ pending entries and bulk processing matters.

הקשר ופירוט

נסקר 2026-06-03 — אין blocker נוכחי. הרצתי reindex ל-73 תקדימים + חילוצים מרובים בלי שמהירות הייתה כאב. YAGNI: לא מבצעים אופטימיזציה מוקדמת. נשאר deferred עם trigger ברור: פסק-דין 200K+ תווים שתוקע את התור, או 50+ פריטים ממתינים. ה-win הזול (concurrency 3→6) דורש אימות headroom של 1.8GB RSS ב-Nautilus לפני — לא עכשיו.

Acceptance Criteria

⚠️ טרם הוגדרו. המשימה הועברה מ-TaskMaster ללא testStrategy.
לפני לקיחתה — להגדיר AC ולסמן status:ready (כלל התבנית: issue ללא AC אינו pickable).

_{הועבר מ-TaskMaster (tag legal-ai, id 14, status היה deferred) ב-2026-08-05. הפניות (#14) בהודעות-commit ישנות מתייחסות למזהה ה-TaskMaster, לא למספר ה-issue הזה.}

## What & Why Halacha extraction on long rulings is slow (5-15 min for typical court ruling, 30-50 min for a 207-chunk appeals committee decision). Root cause: each chunk spawns a separate `claude -p` subprocess (5-10 sec startup overhead each), Hebrew prompts on cold cache run 30-90 sec, and there's no prompt-cache sharing between chunks. Acceleration options to evaluate later when speed becomes a real blocker. Options (each can be combined): 1. Concurrency 3 -> 6 in halacha_extractor.CHUNK_CONCURRENCY. ~2x faster wall-clock. Cost: 6x ~300MB RSS = 1.8GB peak — verify on Nautilus headroom. 2. Larger chunks 12K -> 18-25K chars (CHUNK_TARGET_CHARS in claims_extractor.py / halacha_extractor.py). Fewer waves. Risk: timeout on cold cache (currently 1800s ceiling), and may degrade extraction precision for very long sections. 3. Anthropic SDK direct with 5-min ephemeral prompt caching on the static instruction prefix (already wired the parameter as system= in claude_session.query). Estimated 5-10x faster because cache reads are ~10% of cold cost. Costs ~$0.30-2 per long ruling on Sonnet 4.6. Chair previously rejected this path for ALL traffic ('we work only with claude session'). Compromise: SDK only for the precedent-library corpus build (static, one-time), claude session for live decision drafting (interactive, frequent). 4. Two-tier prompt: a short 'classification' pass with claude -p deciding which chunks contain halachot, then deep extraction only on positive chunks. Could cut total LLM time by 40-60% on rulings with lots of factual chapters. 5. Already implemented (Apr 3, 2026): skip non-extractable sections — only run on chunks where section_type IN (legal_analysis, ruling, conclusion); fallback to all chunks when chunker labels nothing. So that win is already banked. Re-evaluate when: a chair drops a 200K+ char ruling into the queue and the wait becomes painful, OR when the precedent-library has 50+ pending entries and bulk processing matters. ## הקשר ופירוט נסקר 2026-06-03 — אין blocker נוכחי. הרצתי reindex ל-73 תקדימים + חילוצים מרובים בלי שמהירות הייתה כאב. YAGNI: לא מבצעים אופטימיזציה מוקדמת. נשאר deferred עם trigger ברור: פסק-דין 200K+ תווים שתוקע את התור, או 50+ פריטים ממתינים. ה-win הזול (concurrency 3→6) דורש אימות headroom של 1.8GB RSS ב-Nautilus לפני — לא עכשיו. ## Acceptance Criteria > ⚠️ **טרם הוגדרו.** המשימה הועברה מ-TaskMaster ללא `testStrategy`. > לפני לקיחתה — להגדיר AC ולסמן `status:ready` (כלל התבנית: issue ללא AC אינו pickable). --- <sub>הועבר מ-TaskMaster (tag `legal-ai`, id **14**, status היה `deferred`) ב-2026-08-05. הפניות `(#14)` בהודעות-commit ישנות מתייחסות למזהה ה-TaskMaster, לא למספר ה-issue הזה.</sub>

chaim added the area:extraction priority:p3-low status:deferred type:chore labels 2026-08-05 11:01:10 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ezer-mishpati/legal-ai#433