fix(X13): harden court-fetch against browser leaks + reaper for task-master-mcp leak
שלוש שכבות-הגנה נגד דליפת-זיכרון מדפדפנים יתומים, + טיפול בדליפה הגדולה
בפועל בשרת (task-master-mcp).
- camofox_client.py:
- asyncio.wait_for קשיח סביב כל ה-fetch (COURT_FETCH_HARD_TIMEOUT_S=180ש')
— hang → ביטול → async-with tear-down → reap.
- _reap_orphan_browsers(): הורג camoufox-bin יתומים (ppid=1) לפני ואחרי כל
fetch. סדרתיות (INV-CF4) → כל ppid=1 הוא שארית בטוחה.
- scripts/reap_orphan_procs.py: reaper כללי ל-task-master-mcp (~3GB יתומים)
+ camoufox-bin. רק ppid=1; /proc טהור. --dry-run / --loop N.
- scripts/legal-reaper.config.cjs: דמון pm2 (loop 180s, max_memory_restart 100M).
- X13 spec + SCRIPTS.md: תיעוד שכבות-ההגנה.
max_memory_restart בשירות (1.5G) כבר נותן רשת-ביטחון ברמת-התהליך.
Invariants: מקיים INV-CF4 (politeness/serial) — ללא שינוי חוזה.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,8 @@
|
||||
| `eval_gold_bootstrap.py` | python | **FU-5 (GAP-11) — bootstrap ל-gold-set** של הערכת-אחזור ל-`data/eval/gold-set.jsonl`. שני מקורות: `--source citations` (cited==relevant מ-`search_relevance_feedback`; ריק עד שייצברו ציטוטים) ו-`--source known_item` (query=שם-תיק → relevant=עצמו; אות אמיתי היום). Idempotent — שומר שורות `source=chair`, מחדש `bootstrap_*`. דורש POSTGRES. | לפני eval; חוזר כשנצבר ground-truth |
|
||||
| `eval_retrieval.py` | python | **FU-5 (GAP-11, INV-RET4/G8) — harness הערכת-אחזור** — מריץ את מסלול-האחזור בייצור (`search_library`/`search_internal`) על ה-gold-set, מחשב precision@k/recall@k/MRR/nDCG@k (k=5,10), מצרף overall+per-corpus+per-PA ל-`data/eval/eval-report-<ts>.{json,md}` + delta מול `data/eval/baseline.json` (מתעד retrieval_config). `--self-test` בודק את המטריקות offline; `--update-baseline` מאמץ snapshot. **שער-CI במשמעת:** הרץ לפני/אחרי כל שינוי בשכבת-האחזור באותו קונפיג. דורש POSTGRES+VOYAGE_API_KEY. | לפני/אחרי שינוי RRF/k/embedder/rerank |
|
||||
| `legal-court-fetch-service.config.cjs` | pm2/js | **שירות-מארח Tier-1 לאחזור פסקי-דין מנט המשפט (X13)** — 2 apps: (א) `legal-court-fetch-xvfb` (Xvfb :99, צג-וירטואלי ל-Camoufox); (ב) `legal-court-fetch-service` (`python -m legal_mcp.court_fetch_service.server`, bound `10.0.1.1:8771`, Bearer `COURT_FETCH_SHARED_SECRET` מ-`~/.legal-court-fetch-service.env`, `DISPLAY=:99`). מריץ Camoufox דרך חבילת-הפייתון (in-process) כי הקונטיינר לא יכול דפדפן. תלות: `pip install -e "mcp-server[court-fetch]" && python -m camoufox fetch`. אחזור = ניווט→צופה→`GetImages`(X-Requested-With)→PDF, ללא CAPTCHA; כשל→`ok:false`→orchestrator מסלים ל-fallback אנושי. **אומת על עת"מ 46111-12-22 (34 עמ').** מראָה לדפוס `legal-chat-service.config.cjs`. ספ: `docs/spec/X13-court-fetch.md`. התקנה: `pm2 start scripts/legal-court-fetch-service.config.cjs && pm2 save`. בריאות: `curl http://10.0.1.1:8771/health`. | pm2 (host-side) |
|
||||
| `reap_orphan_procs.py` | python | **reaper לתהליכים-יתומים שמרווים את שרת Nautilus** — הורג `task-master-mcp` (Node, מתנפח ל~3GB) ו-`camoufox-bin` (Firefox מ-X13 fetch שקרס) **רק כשהם יתומים (`ppid=1`)** — תהליך עם הורה-חי לעולם לא נוגעים בו. `/proc` טהור, בלי psutil. `--dry-run` (דיווח), `--loop N` (דמון כל N ש'). ראה זיכרון [[project_taskmaster_mcp_memory_leak]]. | דרך `legal-reaper.config.cjs` (pm2) |
|
||||
| `legal-reaper.config.cjs` | pm2/js | **דמון pm2 ל-`reap_orphan_procs.py --loop`** (ברירת-מחדל 180ש', `REAP_INTERVAL_S` לעקיפה). `max_memory_restart 100M` (ה-reaper עצמו לא ידלוף). התקנה: `pm2 start scripts/legal-reaper.config.cjs && pm2 save`. לוגים: `pm2 logs legal-reaper`. | pm2 (host-side) |
|
||||
| `auto-sync-cases.sh` | bash | סנכרון תיקי ערר ל-Gitea — רץ כל דקה | `* * * * *` (cron) |
|
||||
| `backup-db.sh` | bash | גיבוי PostgreSQL יומי ל-`data/backups/` (gzip) | לתזמן: `0 2 * * *` |
|
||||
| `restore-db.sh` | bash | שחזור DB מגיבוי (companion ל-backup-db.sh) | ידני |
|
||||
|
||||
35
scripts/legal-reaper.config.cjs
Normal file
35
scripts/legal-reaper.config.cjs
Normal file
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* pm2 ecosystem entry for legal-reaper — a host-side daemon that periodically
|
||||
* reaps orphaned, runaway processes that saturate the Nautilus box:
|
||||
* - task-master-mcp (Node) orphaned to ppid=1, ballooning to ~3GB each
|
||||
* (memory: project_taskmaster_mcp_memory_leak).
|
||||
* - camoufox-bin (Firefox) leftover from a crashed/killed X13 court fetch.
|
||||
* Only ppid=1 orphans are killed — live, parented processes are never touched.
|
||||
* See scripts/reap_orphan_procs.py for the safety rationale.
|
||||
*
|
||||
* Install (once):
|
||||
* pm2 start /home/chaim/legal-ai/scripts/legal-reaper.config.cjs
|
||||
* pm2 save
|
||||
* Logs:
|
||||
* pm2 logs legal-reaper --lines 50
|
||||
*
|
||||
* Interval defaults to 180s; override with REAP_INTERVAL_S.
|
||||
*/
|
||||
const interval = process.env.REAP_INTERVAL_S || "180";
|
||||
|
||||
module.exports = {
|
||||
apps: [
|
||||
{
|
||||
name: "legal-reaper",
|
||||
cwd: "/home/chaim/legal-ai",
|
||||
script: "/home/chaim/legal-ai/mcp-server/.venv/bin/python",
|
||||
args: `scripts/reap_orphan_procs.py --loop ${interval}`,
|
||||
env: { HOME: "/home/chaim", PYTHONUNBUFFERED: "1" },
|
||||
autorestart: true,
|
||||
max_restarts: 20,
|
||||
restart_delay: 5000,
|
||||
// The reaper itself is tiny and must never be the thing that leaks.
|
||||
max_memory_restart: "100M",
|
||||
},
|
||||
],
|
||||
};
|
||||
119
scripts/reap_orphan_procs.py
Normal file
119
scripts/reap_orphan_procs.py
Normal file
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reap orphaned/runaway processes that saturate the Nautilus box.
|
||||
|
||||
Two known offenders (2026-06-07):
|
||||
1. ``task-master-mcp`` (Node) — spawned by the Claude Code VSCode extension,
|
||||
orphaned to ``ppid=1`` when its session ends, then **balloons to ~3GB
|
||||
each**. They accrue as sessions cycle and exhaust RAM within minutes,
|
||||
risking the OOM-killer hitting Postgres/Paperclip. See memory
|
||||
``project_taskmaster_mcp_memory_leak``.
|
||||
2. ``camoufox-bin`` (Firefox) — the X13 court-fetch browser. A fetch that
|
||||
hangs or is killed mid-flight can leave a stray browser orphaned to
|
||||
``ppid=1``. Serial-only fetching means any ``ppid=1`` camoufox-bin is
|
||||
stale and safe to kill.
|
||||
|
||||
Safety: only processes **orphaned to ``ppid=1``** are reaped — a process still
|
||||
owned by a live parent (an attached MCP server, or a browser a fetch is
|
||||
actively using) is never touched. Pure ``/proc`` parsing, no psutil dependency.
|
||||
|
||||
Usage:
|
||||
python scripts/reap_orphan_procs.py # one pass, print what was reaped
|
||||
python scripts/reap_orphan_procs.py --dry-run # report only
|
||||
python scripts/reap_orphan_procs.py --loop 180 # daemon: reap every 180s
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
|
||||
# Process-name substrings to reap when orphaned (ppid==1).
|
||||
TARGETS = ("task-master-mcp", "camoufox-bin")
|
||||
|
||||
|
||||
def _read(path: str) -> str:
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
return f.read().decode("utf-8", "replace")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def _proc_info(pid: str) -> tuple[int, str, int] | None:
|
||||
"""Return (ppid, cmdline, rss_kb) for a pid, or None if it vanished."""
|
||||
status = _read(f"/proc/{pid}/status")
|
||||
if not status:
|
||||
return None
|
||||
ppid, rss = 0, 0
|
||||
for line in status.splitlines():
|
||||
if line.startswith("PPid:"):
|
||||
try: ppid = int(line.split()[1])
|
||||
except (IndexError, ValueError): pass
|
||||
elif line.startswith("VmRSS:"):
|
||||
try: rss = int(line.split()[1])
|
||||
except (IndexError, ValueError): pass
|
||||
cmd = _read(f"/proc/{pid}/cmdline").replace("\x00", " ").strip()
|
||||
return ppid, cmd, rss
|
||||
|
||||
|
||||
def find_orphans() -> list[tuple[str, str, int]]:
|
||||
"""Return [(pid, cmd, rss_kb)] of ppid==1 processes matching TARGETS."""
|
||||
out = []
|
||||
for pid in os.listdir("/proc"):
|
||||
if not pid.isdigit():
|
||||
continue
|
||||
info = _proc_info(pid)
|
||||
if not info:
|
||||
continue
|
||||
ppid, cmd, rss = info
|
||||
if ppid == 1 and any(t in cmd for t in TARGETS):
|
||||
out.append((pid, cmd, rss))
|
||||
return out
|
||||
|
||||
|
||||
def reap(dry_run: bool = False) -> int:
|
||||
orphans = find_orphans()
|
||||
freed_mb = 0
|
||||
for pid, cmd, rss in orphans:
|
||||
name = next((t for t in TARGETS if t in cmd), cmd[:30])
|
||||
freed_mb += rss // 1024
|
||||
if dry_run:
|
||||
print(f"[dry-run] would reap pid={pid} ({name}) rss={rss//1024}MB", flush=True)
|
||||
continue
|
||||
try:
|
||||
os.kill(int(pid), signal.SIGKILL)
|
||||
print(f"reaped pid={pid} ({name}) rss={rss//1024}MB", flush=True)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
except PermissionError:
|
||||
print(f" permission denied for pid={pid} ({name})", flush=True)
|
||||
if orphans:
|
||||
print(f"{'would free' if dry_run else 'freed'} ~{freed_mb}MB "
|
||||
f"from {len(orphans)} orphan(s)", flush=True)
|
||||
return len(orphans)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Reap orphaned task-master-mcp / camoufox-bin")
|
||||
ap.add_argument("--dry-run", action="store_true", help="report only, kill nothing")
|
||||
ap.add_argument("--loop", type=int, default=0, metavar="SECONDS",
|
||||
help="run forever, reaping every N seconds")
|
||||
args = ap.parse_args()
|
||||
if args.loop:
|
||||
print(f"reaper loop: every {args.loop}s targets={TARGETS}", flush=True)
|
||||
while True:
|
||||
try:
|
||||
reap(args.dry_run)
|
||||
except Exception as e: # never let the daemon die
|
||||
print(f"reap error: {e}", flush=True)
|
||||
time.sleep(args.loop)
|
||||
else:
|
||||
reap(args.dry_run)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user