reextract_metadata / reextract_halachot extract & apply but never cleared
metadata_extraction_requested_at / halacha_extraction_requested_at —
only the bulk worker (process_pending_extractions) did. Result: clicking
"חלץ מטא-דאטה" on the edit sheet (or calling precedent_extract_metadata
directly) left the row stuck in the queue forever, with the UI badge
showing "ממתין לחילוץ" even after extraction succeeded.
Mirror the worker's behaviour: on success ('completed' / 'no_changes' /
'no_halachot'), call db.clear_extraction_request to drain the queue.
Coolify deploy required for the FastAPI container; local MCP server
needs a process restart for the change to take effect (long-running).
Earlier commit afcc481 opened request_metadata_extraction and
request_halacha_extraction to all source kinds — but
list_pending_extraction_requests still hard-filtered to external_upload.
Result: stamping a queue request on an internal_committee row succeeded
silently, but the worker (and the queue badge) never saw it. Even with
the auto-wakeup added in c7132ba the CEO would wake, find 0 pending
items, and exit.
Drop the legacy filter so the queue listing matches the writer side.
Coolify deploy required for the FastAPI container to pick this up.
The "חלץ מטא-דאטה" / "חלץ הלכות" buttons in the UI used to only stamp
the queue (set metadata_extraction_requested_at / halacha_extraction_requested_at)
and rely on a human running `mcp__legal-ai__precedent_process_pending` from
local Claude Code to drain it.
That left the user with an unintuitive two-step flow: click button → run
local MCP tool. Meanwhile, the upload endpoint already does the right
thing — after ingest succeeds it calls `pc_wake_for_precedent_extraction`,
which creates a Paperclip issue, assigns it to the CEO, and wakes them
to run `precedent_process_pending` automatically.
Add the same wakeup call to the manual request-metadata / request-halachot
endpoints. Now clicking the button is sufficient — the CEO picks it up
and drains the queue without manual intervention.
Best-effort: matches the upload flow's failure semantics. The queue stamp
still happens even if the wakeup fails, so the user can fall back to the
manual MCP tool when needed. The wakeup outcome is included in the
response under `wakeup` for observability.
Coolify deploy required for the FastAPI container to pick this up.
When a precedent has not had successful halacha extraction yet, show a
small wand icon between the edit and delete buttons. Clicking it queues
the precedent for the local MCP worker (request-halachot endpoint).
Visibility rule (`needsHalachaExtraction`): show when text extraction is
complete AND halacha status is "pending without requested_at" (never
tried) or "failed" (allow retry). Hide while processing, after
completion, or when already queued — to avoid duplicate requests.
Pairs with the metadata-extract button on the edit sheet.
The "חלץ מטא-דאטה" / "חלץ הלכות" buttons in the UI were returning 404
for any precedent with `source_kind != 'external_upload'`. The original
restriction was meant to keep LLM extraction off internal-committee
imports (their metadata supposedly came from the case file system),
but the same precedent rows can still need re-extraction when ingest
produces broken data — e.g. the corrupted `subject_tags` value
`['[','"','ה','י',...]` that motivated this change (an early ingest
stored a JSON literal into a TEXT[] column, which Postgres split into
single chars).
Two changes here:
1. db.request_metadata_extraction / request_halacha_extraction:
drop the `AND source_kind='external_upload'` filter. The extractor
already preserves user values (only fills empty fields), so this
is safe.
2. precedent_metadata_extractor.extract_and_apply: detect the
character-by-character corruption above and treat it as empty so
the freshly-extracted tags actually replace the broken ones.
Heuristic: 3+ elements where every element is at most 2 chars
(legitimate tags are multi-character Hebrew words).
Coolify deploy required for the FastAPI container to pick this up.
The Hermes Knowledge Curator's hermes-curator.md says it must be able to
read both DOCX and PDF final decisions. The original implementation
hardcoded the .docx extension only. Extend to try .docx → .pdf → .doc →
.rtf → .txt → .md, returning the first match. extractor.extract_text
already supports all six formats, so no extractor changes needed.
If none found, the not_found response now includes the tried_extensions
list so the caller knows what was attempted.
Verified on case 1130-25 (.docx still picked first) and tested via
`curator-cmp mcp test legal-ai`.
The Knowledge Curator (Hermes) couldn't read סופי-{case}.docx because
document_get_text only works on rows in the documents table — the final
file is just a copy in the case's exports/ directory, not a tracked
document. CMP-71 hit this and produced an unproductive interaction
asking the user how to fix the access issue.
Add a new MCP tool that:
- Locates exports/סופי-{case_number}.docx via config.find_case_dir
- Extracts text using the existing extractor service (python-docx based)
- Returns JSON with status + text + page_count + truncation info
- Optional max_chars cap for large decisions
Smoke test on case 1130-25: 400-char preview returns proper Hebrew text
beginning with "לפנינו ערר על החלטת הוועדה המקומית...".
The local MCP server reloads on next Hermes spawn (stdio mode), so the
tool is immediately available — no Coolify deploy needed.
Curator's promptTemplate (DB-stored) updated to use the new tool as the
primary path for reading the final.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The curator's promptTemplate (stored in DB) now teaches Hermes how to
post issue_thread_interactions instead of free-text comments. Three
patterns supported, curator picks per context:
- ask_user_questions for filtering findings (multi-select)
- request_confirmation for accept/reject of a single proposal
- suggest_tasks for proposing follow-up issues
Verified end-to-end on CMP-71: curator hit a real obstacle (couldn't
read the final DOCX from its container) and chose request_confirmation
on its own to ask the user how to proceed — exactly the conversational
behavior we want.
Paperclip auto-wakes the curator with $PAPERCLIP_APPROVAL_ID when the
user responds. The new prompt has a §B branch that handles the second
wake (read response → act → close).
The UI side was already built in d099470 (mirror Paperclip interactions
in case page) — now Hermes-side agents produce interactions too, not
just claude_local agents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingest_final_version uses claude_session internally, which requires the
Claude CLI binary (not present in the legal-ai FastAPI container). The
call always failed with "Claude CLI not found" — caught by try/except
but noisy.
Replace with a static skipped status + comment pointing to the architectural
rule. Run ingest_final_version manually via Claude Code / MCP from the
local host when populating case_law is desired.
The curator wakeup hook remains and works correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous F2 stage in legal-ceo.md fired after the first DOCX export
— too early, since the user often iterates with עריכה-* uploads after
the first export. The true "this is dafna's chosen final" signal is the
"סמן כסופי" button in the UI, which calls api_mark_final.
This commit moves the curator wakeup from CEO's instructions to a
direct hook in api_mark_final:
- web/paperclip_client.py: add CURATOR_AGENTS dict (CMP + CMPA UUIDs)
and wake_curator_for_final() helper. Looks up main case issue,
creates a child issue assigned to the curator, tags plugin_state for
case visibility, and triggers wakeup via Paperclip API.
- web/app.py: api_mark_final now calls workflow_tools.ingest_final_version
(so case_law table finally gets populated for search_decisions) and
pc_wake_curator_for_final. Both are best-effort — failure does not
block marking final.
- legal-ceo.md: remove F2 stage, leave only the agents-table reference
noting the curator runs from api_mark_final.
- hermes-curator.md: update activation description to reflect the new
flow.
Result: curator runs only when chaim deliberately clicks "סמן כסופי",
on the actual final file, with no risk of analyzing a draft that will
later change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds new sub-agent "מנהל ידע" (hermes_local adapter) that runs after
each successful export to analyze the final decision and suggest updates
to skills/decision/SKILL.md and lessons. Read-only on case data, write
only on a single comment per run.
- legal-ceo.md: new stage F2 after F (export). Looks up curator by name
in current company, creates async sub-issue, no waiting. Falls back to
silent skip if no curator configured.
- legal-ceo.md: agents table updated with both curator UUIDs (CMP + CMPA).
- hermes-curator.md: role instructions documenting CMP/CMPA split and
what the curator does/does not do.
Stage 1 POC. End-to-end validated on CMP-68 (case 1130-25) with two
substantive findings on style patterns. CMPA agent created with separate
~/.hermes/profiles/curator-cmpa profile (own MEMORY.md focused on
היטל השבחה / פיצויים).
Known gaps to follow up: curator does not auto-close its issue, does
not auto-persist findings to MEMORY.md, comment attribution falls back
to chaim's user (install-key) — these are tracked separately and do
not block validation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three-layer separation: style learning (style_corpus), appeals-committee decisions
(internal_committee), and court rulings (external_upload).
- SCHEMA_V10: chair_name + district columns on case_law and cases, partial indexes
- create_internal_committee_decision() DB upsert function
- search_precedent_library_semantic() now accepts source_kind/district/chair_name params
- search_precedent_library_hybrid() passes through new params
- services/internal_decisions.py: ingest_internal_decision, migrate_from_style_corpus,
migrate_from_external_corpus (identifies rows via source_type='appeals_committee')
- search_internal_decisions() MCP tool (server.py + tools/search.py)
- internal_decision_migrate() MCP admin tool
- Web endpoints: POST /api/internal-decisions/upload, POST /api/internal-decisions/migrate,
GET /api/internal-decisions
- ingest_final_version auto-ingests finalized decisions into internal corpus
- SKILL.md updated: agents now search internal + external in parallel, present separately
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
paperclip-dev is for maintaining the Paperclip codebase itself — not
relevant to legal work. Removed from all 14 agents (was on CMPA mirror).
paperclip-converting-plans-to-tasks helps decompose a plan into assigned
issues. Useful for the planning-heavy agents (CEO, analyst). Now scoped
to those two — removed from the other 5 in CMPA where it had crept in.
Net effect: zero drift on paperclipai/* skills across all 7 master+mirror
pairs. Verified via the new Agents tab dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two false positives surfaced after the Agents tab went live:
1. status (running/idle/paused) is runtime state, not config — drops in
and out as agents pick up issues. Removed from _DRIFT_FIELDS.
2. desiredSkills compared raw, but local/* and company/* skills carry
per-company hashes/scopes by design (sync_agents_across_companies.py
filters local skills with a warning). Comparing them flags every
master+mirror pair that has any local skill on master.
Now compares only paperclipai/* skills (vendor-shipped, must match).
UI shows an inline note explaining the filter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picks up the new GET /api/admin/paperclip-agents endpoint (Task #29) plus
any other endpoint changes accumulated since the last regeneration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After committing the Paperclip gaps refactor, the .bak-pre-* sentinels
served their purpose. Add a wildcard so future similar backups won't be
tracked. Also ignore data/precedent-library/ (binary PDFs, 11MB) and
data/*.db (sqlite caches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
block-he (פתיחה ניטרלית) was previously emitted only in final decisions.
For interim drafts shown to the chair before ruling, including a neutral
opening helps the chair confirm framing before approving downstream blocks.
Skipped if empty, so legacy cases without block-he are unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the legal-ai ↔ Paperclip integration in line with the official
Paperclip skill. Net effect: HEARTBEAT.md -47% (370→195 lines), all 14
agents on uniform runtime_config + budget + instructionsBundleMode, and
two cross-company helpers replacing manual SQL.
Highlights:
- HEARTBEAT.md refactor: project-specific only, delegates to the official
paperclipai/paperclip skill (loaded per agent). Adds heartbeat-context
fast-path (§1.7) and PAPERCLIP_WAKE_PAYLOAD_JSON shortcut (§1.5).
- Issue Thread Interactions API: legal-ceo.md now uses
ask_user_questions / request_confirmation / suggest_tasks instead of
free-text comments — gives chair structured UI with idempotency keys.
- pc.sh + paperclip_api.pc_request: every API call goes through helpers
that inject Authorization + X-Paperclip-Run-Id (audit trail).
- sync_agents_across_companies.py: master(CMP)→mirror(CMPA) sync via
Paperclip API, idempotent, with --verify and --apply modes.
- skills/new-company-setup: 11-step blueprint distilling all 11 gaps
into a single onboarding runbook for the next company.
- .taskmaster: 12 tasks covering each gap (one already closed: #29).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task #29: surfaces all 14 agents (7 roles × 2 companies) in /settings as
master+mirror pairs with drift detection. Replaces ad-hoc psql + script
inspection with a single dashboard.
Backend: GET /api/admin/paperclip-agents — fetches via Paperclip API
(not direct DB), groups by name, computes drift across model/effort/
timeoutSec/maxTurnsPerRun/skills/runtime_config.heartbeat/budget/status.
Frontend: new AgentsTab card-per-pair with side-by-side compare,
drift highlighting, expandable details (skills list + instructions path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface issue_thread_interactions (ask_user_questions / request_confirmation /
suggest_tasks) directly inside legal-ai's case detail feed so the user can
answer agent prompts without switching to Paperclip's UI.
Backend (FastAPI):
- paperclip_client.py: 4 new helpers — get_issue_interactions (DB),
respond_to_interaction / accept_interaction / reject_interaction (REST).
- app.py: extends GET /api/cases/{case_number}/agents to include
`interactions`, and adds POST /api/cases/{case_number}/agents/interaction-response
routing to /respond, /accept, /reject in Paperclip.
- paperclip_client.py: also pulls existing httpx calls onto the centralized
pc_request helper (paperclip_api.py) for consistent auth + run-id headers.
Frontend (web-ui, Next.js 16 + TanStack Query):
- agents.ts: Interaction / InteractionPayload / InteractionStatus types,
useSubmitInteraction mutation hook (invalidates the activity query).
- agent-activity-feed.tsx: InteractionCard renders radio (single) /
checkbox (multi) for ask_user_questions, accept/reject + reason for
request_confirmation, task selection for suggest_tasks. Resolved
interactions show a read-only summary. Cards are interleaved with
comments by created_at, so the feed reads chronologically.
Paperclip auto-wakes the issue assignee on a successful response
(queueResolvedInteractionContinuationWakeup) — no explicit wakeup needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Radix Tabs defaults dir to 'ltr' if not set explicitly, which broke
RTL inside Tab content (cards flowing left-to-right). Set dir='rtl'
on the Tabs root and translate trigger labels to Hebrew (kept
Paperclip in English as a brand name).
Read-only display of BLOCK_CONFIG from block_writer.py with CREAC role
and JWM functional-purpose annotations per block (sourced from
docs/block-schema.md).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Investigation showed legal-ai container has no INFISICAL_TOKEN and there
is no /legal-ai folder in Infisical — all env vars are stored in Coolify
and injected into os.environ at container start.
- Replace _read_infisical_values with _read_coolify_envs
- New: _coolify_authoritative_value picks among Coolify duplicates
- PATCH writes via Coolify API (upsert by key)
- Drift = Coolify-stored vs container-runtime (common: Coolify edited
without redeploy)
- Response field renamed: infisical_value → coolify_value
- New 'has_duplicates' flag per row when Coolify has multiple entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces stub RegistrationsTab with a full read-only view grouped by client.
Handles all 4 states: loading skeleton, fetch error, host_path_unavailable,
empty list, and populated data with per-registration detail rows.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces stub ToolsTab with a grouped-by-module grid of clickable tool cards.
Adds ToolDetailDrawer (Sheet) showing name, description, module, source_location,
and params_schema for the selected tool.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add drift-badge, env-var-editor, env-var-row components and replace the
environment-tab stub; install shadcn Switch which was missing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extracts Paperclip companies + tag-mappings UI into PaperclipTab component,
adds stub tabs for Environment / Tools / Registrations, and replaces the flat
page.tsx with a shadcn Tabs layout to make room for Tasks 8-10.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add infisical_available flag to _build_env_var_row
- Stabilize error code (no exception text in API response)
- Document raw-comparison safety inline
Adds four helper functions (_infisical_client, _infisical_ctx,
_read_infisical_values, _build_env_var_row) and the /api/settings/mcp/env
endpoint that compares Infisical vs container env vars and reports drift.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- reject non-integer floats in int coerce path
- document masking responsibility on to_public_dict
- use tuple for enum_values (immutable)
- treat empty string as None in normalize_for_compare
Static whitelist of 18 env vars (multimodal, rerank, halacha, general,
credentials, connection) with per-key type coercion, secret masking, and
drift-comparison helpers for the upcoming settings UI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Settings page extension to view and edit MCP server config (env vars,
tools, client registrations) — hybrid edit model: non-secrets editable
through Infisical, secrets read-only with drift detection vs container.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Global search rows linked to /precedents/<case_law_id> but no route
existed, so clicking a result hit a Next 404 and React threw hydration
error #418. New page reads /api/precedent-library/{id} and shows
metadata, summary/headnote/key_quote, subject tags, and the full
halachot roll-up. "ערוך פרטים" opens the existing PrecedentEditSheet
(no duplicate edit UX).
Extracted ExtractedHalachotSection + ReviewStatusPill from the edit
sheet into a shared component so both surfaces render the same block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "ספרייה" tab only exposed approved/total counts in a status pill;
to inspect the actual extracted halachot per case the chair had to use
the global "ממתין לאישור" tab, which only surfaces pending items, or
the MCP tool. Now the per-precedent edit sheet renders a read-only
roll-up of every halacha (approved + pending + rejected) with status
filter tabs and counts. Review actions intentionally stay in the
review tab to avoid duplicate approve/reject UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`rerank.maybe_rerank` calls `base_search(limit=…, **base_kwargs)` on both
the rerank-on and rerank-off paths. Commit 242f668 moved the closure into
hybrid_search.py and renamed its parameter to `limit_inner`, so every call
to `/api/precedent-library/search` raised TypeError 500 regardless of the
VOYAGE_RERANK_ENABLED flag. Sibling `search_documents_hybrid` was unaffected
because it uses `lambda **kw:` which absorbs the kwarg.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Observed 2026-05-03: a `precedent_process_pending(halacha)` run that
chained two precedents (1110/20 → 317/10) succeeded for the first
(9 halachot, 129 chunks) and produced status=`no_halachot` for the
second despite it being a 47KB Supreme Court ruling with rich legal
analysis. A manual single-precedent re-run on 317/10 immediately
extracted 53 halachot. Diagnosis: every chunk's claude_session call
in the back-to-back run silently failed (likely Anthropic rate-limit
storm after the 1110/20 token burn), and the empty list was reported
as "Claude looked and found nothing" — same code path as a real
0-halacha ruling. The user couldn't tell the difference.
Three changes:
1. Surface chunk-level failures (halacha_extractor.py)
`_extract_chunk` now returns `(halachot, succeeded)` so the caller
can count how many chunks crashed. `extract()` uses this to
distinguish:
- `no_halachot` — chunks ran cleanly, Claude found nothing
- `extraction_failed` — ≥50% of chunks crashed AND zero halachot
came back (rate limit, subprocess crash, etc.)
When `extraction_failed`, DB status is left as 'processing' so the
request stays in the queue for the caller to retry — instead of
the old behaviour where it got marked 'completed' and silently
dropped from the queue.
2. Inter-precedent cooldown (precedent_library.py)
`process_pending_extractions` now sleeps 30s between precedents.
Anthropic rate-limits per-org, and back-to-back large rulings
(~4M tokens for 1110/20, immediately followed by another 2-3M)
was the empirical trigger. 30s gives the per-minute counter time
to drain.
3. Auto-retry on extraction_failed (precedent_library.py)
When a precedent comes back as `extraction_failed`, retry once
after a 60s cooldown before giving up. Rate-limit storms are
transient — the manual re-run of 317/10 minutes later succeeded
with 53 halachot and zero chunk failures, confirming a single
retry is sufficient. Only retries `extraction_failed`; never
`no_halachot` (Claude looked and there genuinely is no holding).
The DB status now ends up as 'failed' only after retries are
exhausted, matching the UI's terminal-failure chip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When Paperclip wakes the CEO and the model issues an mcp__legal-ai__*
call within ~10s of session init, Claude Code sometimes returns
"No such tool available" because the legal-ai MCP server hasn't
finished bringing up its tool catalog yet. Observed twice today on
CMPA precedent-extraction wakeups (sessions 9989fbaf and a9c61801);
the agent fell back to bash + .venv/bin/python and finished the work,
but the race needed fixing on the server side.
Three changes that close the window:
1. Lazy schema init (services/db.py + server.py)
`init_schema()` was awaited inside the FastMCP lifespan, blocking
the `initialize`/`tools/list` handshake until ~10 CREATE TABLE IF
NOT EXISTS statements ran. Under contention (two CEOs waking at
once for different companies) this stretched. Now the lifespan
returns immediately and `get_pool()` runs the schema migrations
exactly once on first DB access, guarded by an asyncio.Lock.
tools/list is answered in milliseconds regardless of DB state.
2. Lazy heavy imports
- services/embeddings.py: voyageai (~450ms) loaded only inside
_get_client()
- services/extractor.py: google.cloud.vision (~550ms) loaded only
inside _get_vision_client() and _ocr_with_google_vision()
These two were being imported at module top from
legal_mcp.tools.documents -> services.processor -> services.{
extractor,embeddings}, so the FastMCP server couldn't even start
responding until both finished. Cold start dropped from 2.7s to
1.17s end-to-end (init + tools/list response).
3. Agent-side warmup + retry guidance (.claude/agents/legal-ceo.md)
Even with a fast server, the model can still race on the very
first call. The precedent-extraction section now tells the CEO
to call workflow_status as a warmup probe and to retry after a
short sleep if it sees "No such tool available", before falling
back to the python bypass.
Also expanded the precedent-tool whitelists on the sub-agents that
delegate halacha/library work (commits 4a9a6b7 + 7ee90dc added the
tools to the MCP server but only the CEO got them in its allowed
list). Added to: legal-researcher (full extraction set), legal-analyst
(library_get/list + halacha review), legal-writer (library lookups +
halacha_review), legal-qa (library_get + halacha_review), and the two
that the CEO was already missing (halacha_review, halachot_pending).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
