Chaim
49acde591e
All checks were successful
G12 Leak-Guard / leak-guard (pull_request) Successful in 6s
legal-halacha-drain crashed 29× with asyncpg DeadlockDetectedError. Root cause: every short-lived cron drain re-runs the idempotent schema migrations on startup (get_pool → _run_schema_migrations), and three jobs (metadata-drain, halacha-drain, halacha-supervisor) all fired on the same minute (*/15 / top-of-hour). Two processes running the DDL concurrently took AccessExclusiveLock in opposite order → Postgres killed one with a deadlock. Two-layer fix: - Root cause: wrap _run_schema_migrations in a session-level pg_advisory_lock so only one process applies DDL at a time; concurrent migrators wait instead of deadlocking. DDL body extracted to _apply_schema_ddl. Idempotent, schema unchanged. - Defence-in-depth: give each cron drain a distinct firing minute — metadata :00, supervisor :05, halacha-drain :10, digest :12, court-fetch :17 — so siblings no longer start at the same instant. SCRIPTS.md updated to match. Invariants: G1 (fix at source — the single migration path — not the symptom); G2 (no parallel control path introduced). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
41 lines
1.7 KiB
JavaScript
41 lines
1.7 KiB
JavaScript
/**
|
|
* pm2 ecosystem entry for legal-digest-drain — scheduled (every 2 h) drain of
|
|
* the digest-enrichment queue (X12: "כל יום" yomonim → Sonnet enrichment +
|
|
* embedding + autolink). Migrated from a bare system crontab line to pm2 so it
|
|
* appears in — and is controllable from — the /operations dashboard (run-now /
|
|
* enable / disable) like every other drain.
|
|
*
|
|
* Pattern: cron_restart fires the script on schedule; autorestart:false → runs
|
|
* once and exits (pm2 shows "stopped" between ticks — expected). The script
|
|
* already serialises itself (it self-heals stale 'processing' rows), so no flock
|
|
* is needed under pm2's one-shot model.
|
|
*
|
|
* Requires (host ~/.env via legal_mcp.config): POSTGRES_URL, VOYAGE_API_KEY, and
|
|
* the local `claude` CLI on PATH (the script prepends ~/.local/bin).
|
|
*
|
|
* Install (once):
|
|
* pm2 start /home/chaim/legal-ai/scripts/legal-digest-drain.config.cjs
|
|
* pm2 save
|
|
* Run now (manual): mcp-server/.venv/bin/python scripts/drain_digests.py
|
|
* Schedule override: DIGEST_DRAIN_CRON (default every 2 h at :12).
|
|
*/
|
|
// Minute :12 (not :00) so it never shares a firing minute with the every-hour
|
|
// legal-metadata-drain (:00) — avoids the schema-migration DDL deadlock when
|
|
// sibling drains start at the same instant.
|
|
const cron = process.env.DIGEST_DRAIN_CRON || "12 */2 * * *";
|
|
|
|
module.exports = {
|
|
apps: [
|
|
{
|
|
name: "legal-digest-drain",
|
|
cwd: "/home/chaim/legal-ai",
|
|
script: "/home/chaim/legal-ai/mcp-server/.venv/bin/python",
|
|
args: "scripts/drain_digests.py",
|
|
env: { HOME: "/home/chaim", PYTHONUNBUFFERED: "1" },
|
|
autorestart: false, // one-shot per cron tick
|
|
cron_restart: cron,
|
|
max_memory_restart: "800M",
|
|
},
|
|
],
|
|
};
|