Chaim
49acde591e
All checks were successful
G12 Leak-Guard / leak-guard (pull_request) Successful in 6s
legal-halacha-drain crashed 29× with asyncpg DeadlockDetectedError. Root cause: every short-lived cron drain re-runs the idempotent schema migrations on startup (get_pool → _run_schema_migrations), and three jobs (metadata-drain, halacha-drain, halacha-supervisor) all fired on the same minute (*/15 / top-of-hour). Two processes running the DDL concurrently took AccessExclusiveLock in opposite order → Postgres killed one with a deadlock. Two-layer fix: - Root cause: wrap _run_schema_migrations in a session-level pg_advisory_lock so only one process applies DDL at a time; concurrent migrators wait instead of deadlocking. DDL body extracted to _apply_schema_ddl. Idempotent, schema unchanged. - Defence-in-depth: give each cron drain a distinct firing minute — metadata :00, supervisor :05, halacha-drain :10, digest :12, court-fetch :17 — so siblings no longer start at the same instant. SCRIPTS.md updated to match. Invariants: G1 (fix at source — the single migration path — not the symptom); G2 (no parallel control path introduced). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
58 lines
2.8 KiB
JavaScript
58 lines
2.8 KiB
JavaScript
/**
|
||
* pm2 ecosystem entry for legal-halacha-drain — scheduled drain of the precedent
|
||
* halacha-extraction queue. Halacha extraction stays on claude_session (local
|
||
* CLI, high reasoning quality for holding/ratio) — unlike metadata which moved
|
||
* to Gemini. Extracted halachot land 'pending_review' for the chair's approval
|
||
* gate (INV-G10); this drainer only produces them, it never approves.
|
||
*
|
||
* The drain self-heals orphaned 'processing' rows (precedent_library) and is
|
||
* serialised by a global advisory lock, so overlapping ticks are safe.
|
||
*
|
||
* Pattern: cron_restart fires the script; autorestart:false → one-shot per tick
|
||
* (pm2 shows "stopped" between ticks). Cheap no-op when the queue is empty.
|
||
*
|
||
* NIGHT-WINDOW (23:00–05:00 Israel time): backlog extraction is slow (Opus) and
|
||
* token-heavy, so it runs only off-hours — never competing with daytime
|
||
* interactive work / other agents (avoids the heartbeat timeout+process_lost
|
||
* storm we hit when the CEO drained the whole queue mid-day). Single-case
|
||
* extraction requested by the chair still runs immediately via the CEO; only
|
||
* the bulk backlog is gated to the night.
|
||
*
|
||
* TIMEZONE: pm2's daemon runs in the host TZ (UTC here), so cron strings are
|
||
* UTC. We fire across a UTC *superset* band (20:00–03:00 UTC) that covers the
|
||
* Israel window in BOTH DST states (UTC+3 summer / UTC+2 winter); the script's
|
||
* zoneinfo('Asia/Jerusalem') window-guard then trims to exactly 23:00–05:00
|
||
* and no-ops on the margin ticks. Firing each hour also re-arms the one-shot if
|
||
* a tick died mid-drain — the global advisory lock makes overlap safe.
|
||
*
|
||
* Requires the local ``claude`` CLI + host ~/.env (POSTGRES_URL, etc.).
|
||
*
|
||
* Install (once):
|
||
* pm2 start /home/chaim/legal-ai/scripts/legal-halacha-drain.config.cjs
|
||
* pm2 save
|
||
* Run now (manual, ignores window only if env-overridden):
|
||
* mcp-server/.venv/bin/python scripts/drain_halacha_queue.py
|
||
* Overrides: HALACHA_DRAIN_CRON (UTC cron) · HALACHA_DRAIN_WINDOW_START/_END
|
||
* (Israel hours, default 23/5) · HALACHA_DRAIN_TZ.
|
||
*/
|
||
// UTC band covering Israel 23:00–05:00 across DST; script trims to exact window.
|
||
// Fires at minute :10 (not :00) so it never shares a firing minute with
|
||
// legal-metadata-drain (:00) or legal-halacha-supervisor (:05) — avoids the
|
||
// schema-migration DDL deadlock when sibling drains start at the same instant.
|
||
const cron = process.env.HALACHA_DRAIN_CRON || "10 20,21,22,23,0,1,2,3 * * *";
|
||
|
||
module.exports = {
|
||
apps: [
|
||
{
|
||
name: "legal-halacha-drain",
|
||
cwd: "/home/chaim/legal-ai",
|
||
script: "/home/chaim/legal-ai/mcp-server/.venv/bin/python",
|
||
args: "scripts/drain_halacha_queue.py",
|
||
env: { HOME: "/home/chaim", PYTHONUNBUFFERED: "1" },
|
||
autorestart: false, // one-shot per cron tick
|
||
cron_restart: cron,
|
||
max_memory_restart: "800M",
|
||
},
|
||
],
|
||
};
|