fix(precedents): distinguish silent extraction failure from "no halachot"

Observed 2026-05-03: a `precedent_process_pending(halacha)` run that
chained two precedents (1110/20 → 317/10) succeeded for the first
(9 halachot, 129 chunks) and produced status=`no_halachot` for the
second despite it being a 47KB Supreme Court ruling with rich legal
analysis. A manual single-precedent re-run on 317/10 immediately
extracted 53 halachot. Diagnosis: every chunk's claude_session call
in the back-to-back run silently failed (likely Anthropic rate-limit
storm after the 1110/20 token burn), and the empty list was reported
as "Claude looked and found nothing" — same code path as a real
0-halacha ruling. The user couldn't tell the difference.

Three changes:

1. Surface chunk-level failures (halacha_extractor.py)
   `_extract_chunk` now returns `(halachot, succeeded)` so the caller
   can count how many chunks crashed. `extract()` uses this to
   distinguish:
   - `no_halachot` — chunks ran cleanly, Claude found nothing
   - `extraction_failed` — ≥50% of chunks crashed AND zero halachot
     came back (rate limit, subprocess crash, etc.)
   When `extraction_failed`, DB status is left as 'processing' so the
   request stays in the queue for the caller to retry — instead of
   the old behaviour where it got marked 'completed' and silently
   dropped from the queue.

2. Inter-precedent cooldown (precedent_library.py)
   `process_pending_extractions` now sleeps 30s between precedents.
   Anthropic rate-limits per-org, and back-to-back large rulings
   (~4M tokens for 1110/20, immediately followed by another 2-3M)
   was the empirical trigger. 30s gives the per-minute counter time
   to drain.

3. Auto-retry on extraction_failed (precedent_library.py)
   When a precedent comes back as `extraction_failed`, retry once
   after a 60s cooldown before giving up. Rate-limit storms are
   transient — the manual re-run of 317/10 minutes later succeeded
   with 53 halachot and zero chunk failures, confirming a single
   retry is sufficient. Only retries `extraction_failed`; never
   `no_halachot` (Claude looked and there genuinely is no holding).

The DB status now ends up as 'failed' only after retries are
exhausted, matching the UI's terminal-failure chip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

mcp-server/src/legal_mcp/services/precedent_library.py

@@ -270,6 +270,19 @@ async def reextract_halachot(
     return result
 
 
+# Wait this many seconds between precedents in a multi-precedent run.
+# Anthropic rate-limits across the org, so back-to-back extractions of large
+# rulings (e.g. 129 chunks for one, then 79 for another) can spill the second
+# precedent into a 429 storm. Observed 2026-05-03: 1110/20 succeeded with 9
+# halachot, 317/10 immediately after returned silent no_halachot.
+INTER_PRECEDENT_COOLDOWN_SEC = 30
+
+# How many times to retry a precedent that came back as 'extraction_failed'
+# (i.e. >50% chunks crashed). Each retry uses a longer cooldown.
+PRECEDENT_RETRY_ATTEMPTS = 1
+PRECEDENT_RETRY_COOLDOWN_SEC = 60
+
+
 async def process_pending_extractions(kind: str = "metadata", limit: int = 20) -> dict:
     """Drain the extraction queue (UI-button-stamped requests).
 
@@ -279,6 +292,14 @@ async def process_pending_extractions(kind: str = "metadata", limit: int = 20) -
     tool — picks each stamped row up, runs the extractor, and clears the
     timestamp.
 
+    Sequencing: precedents are processed serially (never in parallel) and
+    each is followed by a short cooldown so the Anthropic rate-limit
+    counter has time to drain before the next big precedent starts. If
+    halacha extraction comes back as ``extraction_failed`` we retry the
+    same precedent once with a longer cooldown — matching the empirical
+    pattern where the second precedent in a back-to-back run gets
+    rate-limited but recovers after a brief pause.
+
     Args:
         kind: 'metadata' or 'halacha'.
         limit: max rows to process this run.
@@ -292,15 +313,46 @@ async def process_pending_extractions(kind: str = "metadata", limit: int = 20) -
     if not pending:
         return {"status": "no_pending", "kind": kind, "processed": 0, "results": []}
 
+    async def _run_once(cid: UUID) -> dict:
+        if kind == "metadata":
+            return await precedent_metadata_extractor.extract_and_apply(cid)
+        return await halacha_extractor.extract(cid)
+
     results: list[dict] = []
     processed = 0
-    for row in pending:
+    for idx, row in enumerate(pending):
+        if idx > 0:
+            await asyncio.sleep(INTER_PRECEDENT_COOLDOWN_SEC)
         cid = UUID(str(row["id"]))
+        attempts = 0
+        result: dict = {}
         try:
-            if kind == "metadata":
-                result = await precedent_metadata_extractor.extract_and_apply(cid)
-            else:
-                result = await halacha_extractor.extract(cid)
+            result = await _run_once(cid)
+            # Retry only on systematic extraction failure (rate-limit storm).
+            # Don't retry on 'no_halachot' — that means Claude looked and
+            # genuinely found nothing.
+            while (
+                result.get("status") == "extraction_failed"
+                and attempts < PRECEDENT_RETRY_ATTEMPTS
+            ):
+                attempts += 1
+                logger.warning(
+                    "process_pending_extractions: %s returned extraction_failed "
+                    "(%d/%d chunks crashed), retry %d/%d after %ds cooldown",
+                    cid,
+                    result.get("failed_chunks", 0),
+                    result.get("total_chunks", 0),
+                    attempts, PRECEDENT_RETRY_ATTEMPTS,
+                    PRECEDENT_RETRY_COOLDOWN_SEC,
+                )
+                await asyncio.sleep(PRECEDENT_RETRY_COOLDOWN_SEC)
+                result = await _run_once(cid)
+
+            # Finalise: success or terminal failure both clear the request
+            # so the queue moves on. (Use 'failed' DB state for terminal
+            # extraction_failed so the UI shows the warning chip.)
+            if kind == "halacha" and result.get("status") == "extraction_failed":
+                await db.set_case_law_halacha_status(cid, "failed")
             await db.clear_extraction_request(cid, kind=kind)
             processed += 1
             results.append({
@@ -309,6 +361,7 @@ async def process_pending_extractions(kind: str = "metadata", limit: int = 20) -
                 "status": result.get("status", "unknown"),
                 "fields": result.get("fields", []),
                 "stored": result.get("stored", 0),
+                "retry_attempts": attempts,
             })
         except Exception as e:
             logger.exception("process_pending_extractions failed for %s: %s", cid, e)
@@ -317,6 +370,7 @@ async def process_pending_extractions(kind: str = "metadata", limit: int = 20) -
                 "case_number": row.get("case_number", ""),
                 "status": "failed",
                 "error": str(e),
+                "retry_attempts": attempts,
             })
             # Don't clear the request — it stays for the next run.