fix(db): serialise schema migrations with an advisory lock + stagger drain crons
All checks were successful
G12 Leak-Guard / leak-guard (pull_request) Successful in 6s
All checks were successful
G12 Leak-Guard / leak-guard (pull_request) Successful in 6s
legal-halacha-drain crashed 29× with asyncpg DeadlockDetectedError. Root cause: every short-lived cron drain re-runs the idempotent schema migrations on startup (get_pool → _run_schema_migrations), and three jobs (metadata-drain, halacha-drain, halacha-supervisor) all fired on the same minute (*/15 / top-of-hour). Two processes running the DDL concurrently took AccessExclusiveLock in opposite order → Postgres killed one with a deadlock. Two-layer fix: - Root cause: wrap _run_schema_migrations in a session-level pg_advisory_lock so only one process applies DDL at a time; concurrent migrators wait instead of deadlocking. DDL body extracted to _apply_schema_ddl. Idempotent, schema unchanged. - Defence-in-depth: give each cron drain a distinct firing minute — metadata :00, supervisor :05, halacha-drain :10, digest :12, court-fetch :17 — so siblings no longer start at the same instant. SCRIPTS.md updated to match. Invariants: G1 (fix at source — the single migration path — not the symptom); G2 (no parallel control path introduced). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -17,9 +17,12 @@
|
||||
* pm2 start /home/chaim/legal-ai/scripts/legal-digest-drain.config.cjs
|
||||
* pm2 save
|
||||
* Run now (manual): mcp-server/.venv/bin/python scripts/drain_digests.py
|
||||
* Schedule override: DIGEST_DRAIN_CRON (default every 2 h at :00).
|
||||
* Schedule override: DIGEST_DRAIN_CRON (default every 2 h at :12).
|
||||
*/
|
||||
const cron = process.env.DIGEST_DRAIN_CRON || "0 */2 * * *";
|
||||
// Minute :12 (not :00) so it never shares a firing minute with the every-hour
|
||||
// legal-metadata-drain (:00) — avoids the schema-migration DDL deadlock when
|
||||
// sibling drains start at the same instant.
|
||||
const cron = process.env.DIGEST_DRAIN_CRON || "12 */2 * * *";
|
||||
|
||||
module.exports = {
|
||||
apps: [
|
||||
|
||||
Reference in New Issue
Block a user