Compare commits
11 Commits
worktree-a
...
worktree-c
| Author | SHA1 | Date | |
|---|---|---|---|
| fbdbc64366 | |||
| 08419e4434 | |||
| 2ebaa82f85 | |||
| 79d9ea55d8 | |||
| 86e66cc5bd | |||
| 42ea1a7c58 | |||
| 10e05700cc | |||
| 11acdac337 | |||
| 4eb3312e9b | |||
| 574998021e | |||
| cc4d757fce |
@@ -143,6 +143,8 @@ tools:
|
||||
**טענות סף** (אם קיימות):
|
||||
חוסר סמכות, שיהוי, התיישנות, אי-מיצוי הליכים, חוסר יריבות, מעשה בית דין — הצג כל אחת עם עמדת שני הצדדים. לכל טענת סף הוסף **עמדת ועדת הערר** (שדה ריק ליו"ר). אם אין — כתוב: "לא זוהו טענות סף."
|
||||
|
||||
**מבנה השדות — זהה לזה של הסוגיות:** כל טענת סף היא H3 משלה, והשדות בתוכה נכתבים בתחילת שורה (`**טענה (claim):**`) — **לא** כפריטי רשימה (`- **טענה (claim):**`) ולא בכותרת מעוטרת. השדה `**עמדת ועדת הערר:**` נכתב בשורה נפרדת והערך מתחתיו. זהו השדה שדפנה עורכת ישירות מה-UI — סטייה מהפורמט מסתירה אותו ואת יתר השדות של אותה טענה מהמסך.
|
||||
|
||||
**תקן ביקורת**: ציין את תקן הביקורת של הוועדה בתיק זה — "הוועדה מפעילה שיקול דעת תכנוני עצמאי" (ברישוי) או "הוועדה בוחנת את תקינות השומה המכרעת" (בהיטל השבחה) או תקן אחר לפי סוג ההליך.
|
||||
|
||||
**מפת דרכים**: לאחר זיהוי טענות הסף ולפני הדיון בסוגיות — כתוב פסקת מפה: "X שאלות עומדות להכרעה: (1)...; (2)...; (3)..." — כדי שהקורא ידע מראש מה לצפות.
|
||||
@@ -347,7 +349,18 @@ wakeup ל-CEO עם `payload.issueId=$PAPERCLIP_TASK_ID` ו-`reason="מנתח מ
|
||||
1. ...
|
||||
|
||||
## 5. טענות סף
|
||||
[אם קיימות — כולל שאלות משפטיות + עמדת ועדת הערר לכל טענה]
|
||||
[אם אין — "לא זוהו טענות סף." אם יש — טענה אחת לכל H3, במבנה הזה:]
|
||||
|
||||
### ס-1 — [כותרת הטענה] ([מי מעלה])
|
||||
|
||||
**טענה (claim):** ...
|
||||
**תשובה (response):** ...
|
||||
**שאלה משפטית:** ...
|
||||
|
||||
**עמדת ועדת הערר:**
|
||||
[ימולא ע"י יו"ר הוועדה]
|
||||
|
||||
### ס-2 — ...
|
||||
|
||||
**תקן ביקורת:** [שיקול דעת עצמאי / בחינת תקינות השומה / אחר]
|
||||
|
||||
|
||||
@@ -820,45 +820,38 @@ ls data/cases/$CASE_NUMBER/documents/research/analysis-and-research.md
|
||||
|
||||
---
|
||||
|
||||
**תבנית issue לכותב ההחלטה — חובה בכל issue שמוקצה לכותב:**
|
||||
**מסמך-ההכוונה לכותב — הפק את התדריך שהיית רוצה לקבל, לא טופס למילוי:**
|
||||
|
||||
כל issue לכותב חייב לכלול את **כל** הסעיפים הבאים. אסור לשלוח issue עם משפט כמו "הועבר לכתיבה" — זה חסר תועלת. הכותב צריך הכל מוכן מראש.
|
||||
כשאתה מעביר תיק לכותב אתה מבצע את **פעולת-ההיסק המרכזית שלך**: להמיר את ניתוח-המנתח + הכרעות-היו"ר למסמך שמאפשר לכותב לנסח החלטה חדה בסגנון דפנה **בלי לחזור אליך**. אל תמלא טופס — הפעל שיפוט משפטי. תדריך טוב:
|
||||
|
||||
- **מוביל בהכרעה ובסוגיה המכריעה** — קבע איזו סוגיה נושאת את התוצאה ומה מייתר את מה, והצב אותה ראשונה.
|
||||
- **בונה כל סוגיה כסילוגיזם** (כלל → עובדות → מסקנה) עם התקדים והמסמך הספציפיים.
|
||||
- **מזהה אדנים עצמאיים** — אם יותר מנימוק אחד מספיק לבדו לתוצאה, אמור זאת מפורשות, כך שנפילת אדן בערעור לא תפיל את ההחלטה.
|
||||
- **בודק עקביות פנימית** — אם שתי הכרעות עלולות להיראות סותרות (למשל דחיית טענה פרשנית אחת וקבלת אחרת), סמן את המתח והסבר את האבחנה לפני שעורך-דין יטען לו.
|
||||
- **עונה לנקודה החזקה של הצד המפסיד** — לא מתעלם ממנה.
|
||||
- **משקלל את הכרעות-היו"ר** ומעביר אותן מילולית.
|
||||
|
||||
**מה התדריך חייב להכיל** (החוזה מול הכותב — אל תשמיט אף רכיב; אל תשלח issue עם "הועבר לכתיבה"):
|
||||
|
||||
```markdown
|
||||
## הנחיות כתיבה — ערר {case_number}
|
||||
|
||||
### 1. תוצאה ומצב
|
||||
- **תוצאה:** {דחייה / קבלה חלקית / קבלה מלאה}
|
||||
- **טיוטה קיימת:** {כן/לא}. אם כן: נתיב מלא לקובץ + הנחיה "קרא את הטיוטה, השתמש בה כבסיס, אל תכתוב מאפס"
|
||||
- **הוראות עריכה מתוך הטיוטה:** {רשימה מדויקת של מה חיים ביקש לשנות — פסקאות, תוכן, placeholders}
|
||||
- **תוצאה:** {דחייה / קבלה חלקית / קבלה מלאה} — עם נימוק קצר ומהי הראיה הניצחת.
|
||||
- **טיוטה קיימת:** {כן/לא}. אם כן: נתיב מלא + "קרא, השתמש כבסיס, אל תכתוב מאפס".
|
||||
- **הוראות עריכה מהטיוטה:** {מה חיים ביקש לשנות — פסקאות, תוכן, placeholders}.
|
||||
|
||||
### 2. סדר סוגיות + מבנה סילוגיסטי
|
||||
לכל סוגיה שצריך לכתוב/לערוך — מבנה סילוגיסטי מלא:
|
||||
|
||||
**סוגיה N: {כותרת}**
|
||||
- סוג ניתוח: {כלל ברור / איזון אינטרסים / מידתיות / שיקול דעת}
|
||||
- כלל (הנחה עליונה): {הוראת תכנית / סעיף חוק / הלכה — ציטוט מדויק}
|
||||
- עובדות (הנחה תחתונה): {העובדות הספציפיות שצריך להחיל — הפנייה למסמך מקור ספציפי}
|
||||
- מסקנה: {מה נובע מהחלת הכלל על העובדות}
|
||||
- תקדימים: {שם פסק דין + מה הוא קובע + למה רלוונטי}
|
||||
- מסמכי מקור: {שמות קבצים ספציפיים ב-data/cases/{case_number}/documents/originals/}
|
||||
### 2. סוגיות — סדר סילוגיסטי, המכריעה מובילה
|
||||
לכל סוגיה: סוג-ניתוח (כלל ברור / איזון / מידתיות / שיקול-דעת) · כלל (ציטוט מדויק של הוראת-תכנית/חוק/הלכה) · עובדות (בהפניה למסמך-מקור ספציפי) · מסקנה · תקדימים (שם + מה קובע + רלוונטיות) · מסמכי-מקור (ב-data/cases/{case_number}/documents/originals/). סמן אדנים עצמאיים, מוקשי-עקביות ומענה לצד המפסיד היכן שהם קיימים.
|
||||
|
||||
### 3. טיפול בטענות
|
||||
| # | טענה | טיפול | סוגיה |
|
||||
|---|------|-------|-------|
|
||||
| 1 | {טענה} | דיון מלא / קיבוץ / דילוג | {באיזו סוגיה} |
|
||||
...
|
||||
טבלה: # | טענה | טיפול (דיון מלא / קיבוץ / דילוג) | סוגיה.
|
||||
|
||||
### 4. chair directions
|
||||
- העתק מלא של עמדות הוועדה מ-analysis-and-research.md (או הפנייה: "קרא get_chair_directions").
|
||||
- **עטוף את ההעתק המילולי בתגית `<chair_directions>…</chair_directions>`** — כך הכותב מבחין בין הוראות-היו"ר המילוליות לבין הערות ה-CEO, ואינו דורס אותן. בתוך התגית: טקסט מילולי בלבד, בלי פרפרזה.
|
||||
### 4. הנחיות-היו"ר
|
||||
העתק מילולי של עמדות-הוועדה מ-analysis-and-research.md (או "קרא get_chair_directions"), **עטוף ב-`<chair_directions>…</chair_directions>`** — טקסט מילולי בלבד בלי פרפרזה, כדי שהכותב לא ידרוס אותן.
|
||||
|
||||
### 5. הנחיות סגנון
|
||||
- ניטרליות: בלוק ו = עובדות בלבד, בלי ציטוטים מצדדים
|
||||
- ללא כפילות: בלוק י מפנה לבלוקים קודמים
|
||||
- טענות מקוריות: בלוק ז = כתבי טענות מקוריים
|
||||
- אורך מינימלי לדיון: 1,500 מילים לבלוק י
|
||||
- פסיקה: חובה לצטט לפחות 3 תקדימים בדיון
|
||||
ניטרליות (בלוק ו = עובדות בלבד, בלי ציטוטי-צדדים) · ללא כפילות (בלוק י מפנה לקודמים) · טענות מקוריות (בלוק ז) · דיון ≥ 1,500 מילים · ≥ 3 תקדימים בדיון.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -23,8 +23,10 @@ from pathlib import Path
|
||||
from legal_mcp import config
|
||||
from legal_mcp.services import db, embeddings, claude_session, audit, storage
|
||||
from legal_mcp.services.lessons import (
|
||||
ANTI_PATTERNS as _ANTI_PATTERNS,
|
||||
OUTCOME_LABELS_HE,
|
||||
PRACTICE_AREA_OVERRIDES,
|
||||
anti_pattern_directive,
|
||||
canonical_outcome,
|
||||
get_content_checklist,
|
||||
get_methodology_summary,
|
||||
@@ -369,6 +371,7 @@ async def write_block(
|
||||
block_id: str,
|
||||
instructions: str = "",
|
||||
effort_override: str | None = None,
|
||||
model_override: str | None = None,
|
||||
) -> dict:
|
||||
"""כתיבת בלוק יחיד בהחלטה.
|
||||
|
||||
@@ -381,6 +384,12 @@ async def write_block(
|
||||
THIS call only — used by the #208 model/effort calibration harness
|
||||
to A/B efforts without mutating the pinned defaults. Production
|
||||
callers leave it None and get the deterministic per-block effort.
|
||||
model_override: optional per-call generation model id (e.g.
|
||||
"claude-opus-5"). Same contract as effort_override — the #208
|
||||
harness A/Bs MODELS without mutating the pinned GENERATION_MODEL.
|
||||
Pass the BASE id only: the 1M-context escalation (#216) is applied
|
||||
on top automatically for large prompts, so an override never
|
||||
silently loses the 1M window. Production callers leave it None.
|
||||
|
||||
Returns:
|
||||
dict עם content, word_count, block_id, generation_type
|
||||
@@ -468,6 +477,12 @@ async def write_block(
|
||||
if instructions:
|
||||
prompt += f"\n\n## הנחיות נוספות:\n{instructions}"
|
||||
|
||||
# LAST in the prompt, deliberately (see lessons.anti_pattern_directive): the
|
||||
# same canonical rule already appears inside style_context, but ~47K chars
|
||||
# deep, where it measurably fails to bind. Restating it here is the only
|
||||
# change the A/B isolated as effective — so nothing may be appended after it.
|
||||
prompt += "\n\n" + anti_pattern_directive()
|
||||
|
||||
# Block י requires approved direction
|
||||
if block_id == "block-yod":
|
||||
dir_doc = (decision or {}).get("direction_doc") or {}
|
||||
@@ -478,7 +493,12 @@ async def write_block(
|
||||
# escalate to the 1M-context build (`[1m]`) instead of failing the block —
|
||||
# block-yod legitimately carries the whole case as source-context. The 400K
|
||||
# ceiling was an artifact of the old 200K-only build, NOT a model limit.
|
||||
gen_model = GENERATION_MODEL_1M if len(prompt) > _CTX_1M_THRESHOLD_CHARS else GENERATION_MODEL
|
||||
# model_override (#208 harness) swaps the BASE id only — the 1M decision below
|
||||
# still applies, so an A/B'd model keeps the same context-window behaviour as
|
||||
# the pinned default instead of silently falling back to the 200K build.
|
||||
_base_model = model_override or GENERATION_MODEL
|
||||
_model_1m = GENERATION_MODEL_1M if _base_model == GENERATION_MODEL else f"{_base_model}[1m]"
|
||||
gen_model = _model_1m if len(prompt) > _CTX_1M_THRESHOLD_CHARS else _base_model
|
||||
|
||||
# Final guard: even the 1M build is finite (~2M Hebrew chars of input). Cap at
|
||||
# 1.5M chars (~750K tokens) to leave room for output + a safety margin under 1M.
|
||||
@@ -1107,6 +1127,16 @@ async def _build_style_context(practice_area: str = "") -> str:
|
||||
# ── למידה מצטברת (T15) — עריכות היו"ר ב-/methodology + לקחי /training ──
|
||||
# גובר על ברירות-המחדל לעיל. כך כל מה שלמדנו עד היום מגיע לכותב.
|
||||
learned: list[str] = []
|
||||
# The canonical anti-patterns are rendered UNCONDITIONALLY, before any DB
|
||||
# call. They used to be produced inside the overrides loop below — so a
|
||||
# failure on an EARLIER category (e.g. golden_ratios) aborted the loop and
|
||||
# dropped the style invariants from the prompt silently, with only a generic
|
||||
# "overrides not loaded" warning to show for it (§6). A chair-override
|
||||
# outage must not be able to un-teach Dafna's structural style.
|
||||
learned.append("\n**אנטי-דפוסים (להימנע) — כתוב נרטיב משפטי רציף; הימנע מ:**")
|
||||
for ap in _ANTI_PATTERNS:
|
||||
learned.append(f"- {ap['note']}")
|
||||
|
||||
try:
|
||||
for cat, label in (
|
||||
("golden_ratios", "יחסי-זהב (אחוזי-סעיפים)"),
|
||||
@@ -1125,10 +1155,8 @@ async def _build_style_context(practice_area: str = "") -> str:
|
||||
# corrects them, and drafts keep emitting them (the gap that left
|
||||
# 8137 with 28 hits). Chair additions layer on top; they never
|
||||
# remove the canonical ones.
|
||||
from legal_mcp.services.lessons import ANTI_PATTERNS as _ANTI
|
||||
learned.append(f"\n**{label} — כתוב נרטיב משפטי רציף; הימנע מ:**")
|
||||
for ap in _ANTI:
|
||||
learned.append(f"- {ap['note']}")
|
||||
# The canonical list is already rendered above, outside this try —
|
||||
# here we only layer the chair's ADDITIONS on top of it.
|
||||
for k, v in (ov or {}).items():
|
||||
learned.append(f"- (יו\"ר) {k}: {json.dumps(v, ensure_ascii=False)}")
|
||||
continue
|
||||
@@ -1264,6 +1292,12 @@ async def get_block_context(case_id: UUID, block_id: str, instructions: str = ""
|
||||
if instructions:
|
||||
formatted_prompt += f"\n\n## הנחיות נוספות:\n{instructions}"
|
||||
|
||||
# Same closing directive, same position, same canonical source as write_block.
|
||||
# This is the EXTERNAL-writer path (legal-writer agent) — if the rule were
|
||||
# applied only in write_block, agent-written blocks would keep emitting the
|
||||
# anti-patterns and the two writers would drift apart (G2).
|
||||
formatted_prompt += "\n\n" + anti_pattern_directive()
|
||||
|
||||
# Block י requires approved direction
|
||||
if block_id == "block-yod":
|
||||
dir_doc = (decision or {}).get("direction_doc") or {}
|
||||
|
||||
@@ -59,6 +59,28 @@ ANTI_PATTERNS: list[dict] = [
|
||||
"note": "רשימות תבליטים באנליזה — דפנה כותבת נרטיב רציף"},
|
||||
]
|
||||
|
||||
|
||||
def anti_pattern_directive() -> str:
|
||||
"""The closing style directive, rendered from ANTI_PATTERNS (the same list
|
||||
style_distance scores against — one source, two renderings, not two rules).
|
||||
|
||||
WHY THIS EXISTS SEPARATELY FROM the style-context rendering: the rule was
|
||||
already reaching the writer, buried ~47K chars deep inside style_context,
|
||||
and drafts kept emitting the very patterns it forbids. A measured A/B over
|
||||
the signed finals (9 cases, 60 generations, 2026-07-28) showed that the SAME
|
||||
rule restated at the END of the assembled prompt cuts anti-pattern hits by
|
||||
72–93% on both blocks and both models:
|
||||
|
||||
block-vav opus-4-8 1.75 → 0.12 | opus-5 2.25 → 0.62
|
||||
block-zayin opus-4-8 4.57 → 0.43 | opus-5 4.43 → 0.43
|
||||
|
||||
So this is a POSITION fix, not a new instruction. Keep it last in the prompt.
|
||||
"""
|
||||
lines = ["## כלל-סגנון מחייב (גובר על כל דוגמה בהקשר שלמעלה)",
|
||||
"כתוב נרטיב משפטי רציף בלבד — פסקאות שלמות. אסור:"]
|
||||
lines += [f"- {ap['note']}" for ap in ANTI_PATTERNS]
|
||||
return "\n".join(lines)
|
||||
|
||||
# ── Paragraph length guidance (word counts) ────────────────────────
|
||||
|
||||
PARAGRAPH_LENGTHS = {
|
||||
|
||||
@@ -43,8 +43,18 @@ SUBSECTION_RE = re.compile(r"^###\s+(.+?)$", re.MULTILINE)
|
||||
# Matches "**LABEL:**" field markers — handles both inline and block variants:
|
||||
# "**עמדת המבקשת:** Some text on same line"
|
||||
# "**שאלות משפטיות:**\n1. First question"
|
||||
# and both the bare and the list-item form, because the analyst agent writes
|
||||
# threshold claims as a bullet list ("- **עמדת ועדת הערר:**") while it writes
|
||||
# issues bare. Group 1 is the list marker (or None), group 2 is the label.
|
||||
# The label itself must not contain ** or newlines.
|
||||
FIELD_LABEL_RE = re.compile(r"^\*\*([^\n*]+?):\*\*[ \t]*", re.MULTILINE)
|
||||
FIELD_LABEL_RE = re.compile(
|
||||
r"^([ \t]*(?:[-*+]|\d+[.)])[ \t]+)?\*\*([^\n*]+?):\*\*[ \t]*",
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
# Terminators that end a field's content even without a following field label:
|
||||
# a heading, or a horizontal rule closing the subsection.
|
||||
FIELD_TERMINATOR_RE = re.compile(r"^(?:#{2,}[ \t]|[ \t]*---[ \t]*$)", re.MULTILINE)
|
||||
|
||||
# Matches the case number in the H1
|
||||
CASE_NUMBER_RE = re.compile(r"#\s*ניתוח.*?ערר\s+([\d/\-]+)", re.MULTILINE)
|
||||
@@ -53,6 +63,17 @@ CASE_NUMBER_RE = re.compile(r"#\s*ניתוח.*?ערר\s+([\d/\-]+)", re.MULTILIN
|
||||
DATE_RE = re.compile(r"^תאריך:\s*(.+?)\s*$", re.MULTILINE)
|
||||
|
||||
|
||||
def _is_chair_label(label: str) -> bool:
|
||||
"""Is this field label the chair-position field?
|
||||
|
||||
Matches on prefix, not equality, because the analyst sometimes decorates the
|
||||
label with a parenthetical — "עמדת ועדת הערר (הכוונת יו"ר 24.6)". Requiring
|
||||
equality made the reader treat those as ordinary fields and the writer append
|
||||
a second, duplicate block instead of updating the existing one.
|
||||
"""
|
||||
return label.strip().startswith(CHAIR_POSITION_LABEL)
|
||||
|
||||
|
||||
def _is_placeholder(text: str) -> bool:
|
||||
"""Check if a field value is one of the placeholder strings (empty)."""
|
||||
stripped = text.strip()
|
||||
@@ -135,7 +156,7 @@ def _extract_fields(text: str) -> list[dict]:
|
||||
|
||||
fields = []
|
||||
for i, m in enumerate(matches):
|
||||
label = m.group(1).strip()
|
||||
label = m.group(2).strip()
|
||||
content_start = m.end()
|
||||
content_end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
||||
content = text[content_start:content_end].strip()
|
||||
@@ -164,11 +185,17 @@ def _build_subsection_dict(
|
||||
parts = title.split(": ", 1)
|
||||
display_title = parts[1] if len(parts) > 1 else title
|
||||
|
||||
# Only the *first* chair-position field is the editable one — the same one
|
||||
# update_chair_position writes to. A subsection that carries more than one
|
||||
# (e.g. an H3 covering "סוגיות 4–6") keeps the extras as ordinary fields
|
||||
# rather than silently reading back a different field than the one saved.
|
||||
chair_position = ""
|
||||
chair_seen = False
|
||||
regular_fields = []
|
||||
for f in fields:
|
||||
if f["label"] == CHAIR_POSITION_LABEL:
|
||||
if not chair_seen and _is_chair_label(f["label"]):
|
||||
chair_position = _normalize_chair_position(f["content"])
|
||||
chair_seen = True
|
||||
else:
|
||||
regular_fields.append(f)
|
||||
|
||||
@@ -311,6 +338,46 @@ def _find_subsection_by_id(
|
||||
return None
|
||||
|
||||
|
||||
def _split_trailing_rule(body: str) -> tuple[str, str]:
|
||||
"""Split a subsection body into (content, trailing "---" separator).
|
||||
|
||||
Returns ("<body>", "") when the subsection has no closing rule.
|
||||
"""
|
||||
m = re.search(r"\n[ \t]*---[ \t]*\s*\Z", body)
|
||||
if not m:
|
||||
return body, ""
|
||||
return body[: m.start()], body[m.start() :]
|
||||
|
||||
|
||||
def _chair_field_span(body: str) -> tuple[int, int] | None:
|
||||
"""Locate the chair-position field's *content* range in a subsection body.
|
||||
|
||||
Returns (content_start, content_end) — the slice the chair's text occupies,
|
||||
excluding the "**LABEL:**" marker itself so the marker line (and any list
|
||||
bullet in front of it) survives an update untouched. Returns None when the
|
||||
subsection has no chair-position field yet.
|
||||
|
||||
Field boundaries come from FIELD_LABEL_RE — the same definition the reader
|
||||
uses — so what update writes is exactly what parse reads back (G2: one
|
||||
definition of a field, not two that can drift).
|
||||
"""
|
||||
matches = list(FIELD_LABEL_RE.finditer(body))
|
||||
for i, m in enumerate(matches):
|
||||
if not _is_chair_label(m.group(2)):
|
||||
continue
|
||||
content_start = m.end()
|
||||
if i + 1 < len(matches):
|
||||
content_end = matches[i + 1].start()
|
||||
else:
|
||||
content_end = len(body)
|
||||
# A heading or closing rule ends the field even without a next label.
|
||||
term = FIELD_TERMINATOR_RE.search(body, content_start, content_end)
|
||||
if term:
|
||||
content_end = term.start()
|
||||
return content_start, content_end
|
||||
return None
|
||||
|
||||
|
||||
def update_chair_position(
|
||||
file_path: Path, section_id: str, new_text: str
|
||||
) -> dict[str, Any]:
|
||||
@@ -330,40 +397,62 @@ def update_chair_position(
|
||||
|
||||
_abs_start, _abs_end, subsection_body = found
|
||||
|
||||
# Find the "**עמדת ועדת הערר:**" label within this subsection
|
||||
label_pattern = re.compile(
|
||||
r"(\*\*" + re.escape(CHAIR_POSITION_LABEL) + r":\*\*)\s*\n?([^*]*?)(?=\n\*\*|\n##|\n---|\Z)",
|
||||
re.DOTALL,
|
||||
)
|
||||
m = label_pattern.search(subsection_body)
|
||||
if not m:
|
||||
# Label not present — append it at the end of the subsection
|
||||
# (just before the trailing --- if any)
|
||||
new_block = f"\n\n**{CHAIR_POSITION_LABEL}:**\n{new_text.strip()}\n"
|
||||
new_subsection = subsection_body.rstrip() + new_block
|
||||
new_content = content[:_abs_start] + new_subsection + content[_abs_end:]
|
||||
span = _chair_field_span(subsection_body)
|
||||
body_text = new_text.strip() or CHAIR_POSITION_PLACEHOLDERS[0]
|
||||
|
||||
if span is None:
|
||||
# Label not present — append it at the end of the subsection,
|
||||
# before a trailing horizontal rule if there is one.
|
||||
head, tail = _split_trailing_rule(subsection_body)
|
||||
new_block = f"\n\n**{CHAIR_POSITION_LABEL}:**\n{body_text}\n"
|
||||
new_subsection = head.rstrip() + new_block + tail
|
||||
else:
|
||||
# Replace the existing content of the chair_position field
|
||||
replacement = f"{m.group(1)}\n{new_text.strip() if new_text.strip() else CHAIR_POSITION_PLACEHOLDERS[0]}\n"
|
||||
# Replace the existing content of the chair_position field, keeping the
|
||||
# label line exactly as written (including any list marker) so the file
|
||||
# structure the analyst produced is preserved.
|
||||
content_start, content_end = span
|
||||
new_subsection = (
|
||||
subsection_body[: m.start()] + replacement + subsection_body[m.end():]
|
||||
subsection_body[:content_start].rstrip("\r\n \t")
|
||||
+ f"\n{body_text}\n"
|
||||
+ subsection_body[content_end:]
|
||||
)
|
||||
new_content = content[:_abs_start] + new_subsection + content[_abs_end:]
|
||||
|
||||
new_content = content[:_abs_start] + new_subsection + content[_abs_end:]
|
||||
|
||||
# Atomic write
|
||||
tmp_path = file_path.with_suffix(file_path.suffix + ".tmp")
|
||||
tmp_path.write_text(new_content, encoding="utf-8") # noqa: STG1 — atomic .tmp; in-place edit, S3 re-sync in Phase-2 read-wiring
|
||||
os.replace(tmp_path, file_path)
|
||||
|
||||
preview = new_text.strip()[:120]
|
||||
# Read-after-write: a position the parser cannot read back is not saved,
|
||||
# however cleanly the write itself succeeded. Reporting success here is what
|
||||
# let a whole class of format drift hide behind a green "נשמר" in the UI.
|
||||
stored = _stored_chair_position(file_path, section_id)
|
||||
expected = _normalize_chair_position(new_text)
|
||||
if stored != expected:
|
||||
raise RuntimeError(
|
||||
f"העמדה נכתבה ל-{file_path.name} אך לא נקראה בחזרה עבור {section_id} "
|
||||
f"— ככל הנראה מבנה השדה בקובץ חורג מהתבנית הצפויה"
|
||||
)
|
||||
|
||||
return {
|
||||
"saved": True,
|
||||
"section_id": section_id,
|
||||
"preview": preview,
|
||||
"position": stored,
|
||||
"preview": stored[:120],
|
||||
"timestamp": datetime.now(IL_TZ).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
def _stored_chair_position(file_path: Path, section_id: str) -> str:
|
||||
"""Re-parse the file and return the chair position now stored for a section."""
|
||||
parsed = parse(file_path)
|
||||
for item in parsed.get("threshold_claims", []) + parsed.get("issues", []):
|
||||
if item["id"] == section_id:
|
||||
return item.get("chair_position", "") or ""
|
||||
return ""
|
||||
|
||||
|
||||
# ── Chair directions extraction (for downstream agents) ─────────
|
||||
|
||||
|
||||
|
||||
@@ -176,7 +176,13 @@ def block_distance_to_final(
|
||||
outcome = canonical_outcome(outcome)
|
||||
diff = compute_diff_stats(regenerated_text or "", final_section_text or "")
|
||||
change_percent = diff["change_percent"]
|
||||
anti_total = count_anti_patterns(regenerated_text or "")["total"]
|
||||
anti = count_anti_patterns(regenerated_text or "")
|
||||
anti_total = anti["total"]
|
||||
# Per-pattern breakdown, not just the total: a calibration run that only
|
||||
# reports "anti=4" cannot tell you WHICH rule was broken, so it cannot say
|
||||
# what to fix. (Diagnosing the 2026-07-28 model A/B needed exactly this and
|
||||
# had to fall back on inference.)
|
||||
anti_by_pattern = {name: h["count"] for name, h in anti["by_pattern"].items()}
|
||||
|
||||
section = _BLOCK_TO_SECTION.get(block_id)
|
||||
regen_words = len((regenerated_text or "").split())
|
||||
@@ -205,6 +211,7 @@ def block_distance_to_final(
|
||||
"final_words": final_words,
|
||||
"change_percent": change_percent,
|
||||
"anti_pattern_total": anti_total,
|
||||
"anti_by_pattern": anti_by_pattern,
|
||||
"golden_ratio_deviation_pp": ratio_dev,
|
||||
"distance": distance,
|
||||
}
|
||||
|
||||
60
mcp-server/tests/test_anti_pattern_directive.py
Normal file
60
mcp-server/tests/test_anti_pattern_directive.py
Normal file
@@ -0,0 +1,60 @@
|
||||
"""The style invariants must actually REACH the writer.
|
||||
|
||||
Both tests here cover defects found by the 2026-07-28 model×prompt A/B over the
|
||||
signed finals: the canonical anti-patterns were present in the prompt but buried
|
||||
~47K chars into style_context (where they measurably failed to bind), and they
|
||||
were rendered inside a try/except that an unrelated DB failure could abort.
|
||||
"""
|
||||
import pytest
|
||||
|
||||
from legal_mcp.services import block_writer
|
||||
from legal_mcp.services.lessons import ANTI_PATTERNS, anti_pattern_directive
|
||||
|
||||
|
||||
def test_directive_renders_every_canonical_anti_pattern():
|
||||
"""One source, two renderings — the directive may not drift from the list
|
||||
style_distance scores against."""
|
||||
text = anti_pattern_directive()
|
||||
for ap in ANTI_PATTERNS:
|
||||
assert ap["note"] in text, f"missing anti-pattern in directive: {ap['name']}"
|
||||
|
||||
|
||||
def test_both_writer_paths_append_the_directive_last():
|
||||
"""write_block (in-process) and get_block_context (legal-writer agent) must
|
||||
both close with the directive — otherwise the two writers drift (G2)."""
|
||||
import inspect
|
||||
src = inspect.getsource(block_writer)
|
||||
for fn in ("async def write_block(", "async def get_block_context("):
|
||||
start = src.index(fn)
|
||||
# bound the search to this function: up to the next top-level def
|
||||
rest = src[start + len(fn):]
|
||||
nxt = rest.find("\nasync def ")
|
||||
body = rest[: nxt if nxt != -1 else len(rest)]
|
||||
assert "anti_pattern_directive()" in body, f"{fn} does not append the style directive"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_style_context_keeps_anti_patterns_when_overrides_fail(monkeypatch):
|
||||
"""A chair-override outage must not silently un-teach the structural style.
|
||||
|
||||
Regression: the canonical list used to be emitted inside the overrides loop,
|
||||
so a throw on an EARLIER category (golden_ratios) dropped it entirely.
|
||||
"""
|
||||
async def _boom(*a, **k):
|
||||
raise RuntimeError("methodology table unavailable")
|
||||
|
||||
async def _empty(*a, **k):
|
||||
return []
|
||||
|
||||
# Every DB accessor this function touches is stubbed — the test must not open
|
||||
# a real connection (a live pool here leaks across the shared event loop and
|
||||
# breaks unrelated tests later in the run).
|
||||
monkeypatch.setattr(block_writer.db, "get_style_patterns", _empty)
|
||||
monkeypatch.setattr(block_writer.db, "get_methodology_overrides", _boom)
|
||||
monkeypatch.setattr(block_writer.db, "get_recent_decision_lessons", _empty)
|
||||
|
||||
ctx = await block_writer._build_style_context("היטל השבחה")
|
||||
|
||||
assert "נרטיב משפטי רציף" in ctx
|
||||
for ap in ANTI_PATTERNS:
|
||||
assert ap["note"] in ctx, f"anti-pattern dropped on override failure: {ap['name']}"
|
||||
237
mcp-server/tests/test_research_md_chair_position.py
Normal file
237
mcp-server/tests/test_research_md_chair_position.py
Normal file
@@ -0,0 +1,237 @@
|
||||
"""Chair-position round-trip in analysis-and-research.md.
|
||||
|
||||
Regression cover for the reader/writer asymmetry that made "עמדת ועדת הערר"
|
||||
appear to save on threshold claims and then vanish on refresh: the writer
|
||||
matched the label anywhere on a line, the reader only at the start of one, and
|
||||
the analyst agent writes threshold claims as a bullet list
|
||||
("- **עמדת ועדת הערר:**") while it writes issues flush-left.
|
||||
|
||||
The invariant these tests pin down: whatever update_chair_position writes,
|
||||
parse() must read back — for every label form that appears in the corpus.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from legal_mcp.services import research_md
|
||||
|
||||
|
||||
BULLETED = """# ניתוח משפטי — ערר 1069-04-26
|
||||
תאריך: 2026-08-02
|
||||
|
||||
## 5. טענות סף
|
||||
|
||||
### ס-1 — אי-מילוי תנאי ההפקדה
|
||||
- **טענה (claim):** התכנית אושרה מבלי שמולאו תנאי ההפקדה.
|
||||
- **שאלה משפטית:** האם הסטת התנאים פוגמת בחוקיות?
|
||||
- **עמדת ועדת הערר:** [ימולא ע"י יו"ר הוועדה]
|
||||
|
||||
### ס-2 — זכות עמידה
|
||||
- **טענה (claim):** לעוררים אין אינטרס מוגן.
|
||||
- **עמדת ועדת הערר:** [ימולא ע"י יו"ר הוועדה]
|
||||
|
||||
## 6. סוגיות להכרעה
|
||||
|
||||
### סוגיה 1: סמכות לפי 62א(א)(11)
|
||||
|
||||
**ניתוח:**
|
||||
- הכלל החל: ...
|
||||
|
||||
**עמדת ועדת הערר:** [ימולא ע"י יו"ר הוועדה]
|
||||
|
||||
---
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def analysis_file(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "analysis-and-research.md"
|
||||
path.write_text(BULLETED, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _positions(path: Path) -> dict[str, str]:
|
||||
parsed = research_md.parse(path)
|
||||
items = parsed["threshold_claims"] + parsed["issues"]
|
||||
return {item["id"]: item["chair_position"] for item in items}
|
||||
|
||||
|
||||
# ── the reported bug ────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_bulleted_threshold_position_survives_a_reload(analysis_file: Path) -> None:
|
||||
"""The bug as chaim hit it: saved on a threshold claim, gone on refresh."""
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", "עמדתנו היא שהטענה נדחית.")
|
||||
|
||||
assert _positions(analysis_file)["threshold_1"] == "עמדתנו היא שהטענה נדחית."
|
||||
|
||||
|
||||
def test_flush_left_issue_position_still_round_trips(analysis_file: Path) -> None:
|
||||
"""The surface that already worked must keep working."""
|
||||
research_md.update_chair_position(analysis_file, "issue_1", "יש לקבל את הערר בנקודה זו.")
|
||||
|
||||
assert _positions(analysis_file)["issue_1"] == "יש לקבל את הערר בנקודה זו."
|
||||
|
||||
|
||||
def test_bulleted_threshold_fields_are_parsed_at_all(analysis_file: Path) -> None:
|
||||
"""Bulleted labels were invisible to the reader — the whole claim looked empty."""
|
||||
threshold = research_md.parse(analysis_file)["threshold_claims"]
|
||||
|
||||
labels = [f["label"] for f in threshold[0]["fields"]]
|
||||
assert "טענה (claim)" in labels
|
||||
assert "שאלה משפטית" in labels
|
||||
# the chair field is surfaced separately, never as a regular field
|
||||
assert not any(research_md._is_chair_label(label) for label in labels)
|
||||
|
||||
|
||||
# ── writer/reader agreement on content boundaries ───────────────
|
||||
|
||||
|
||||
def test_position_containing_markdown_is_not_truncated(analysis_file: Path) -> None:
|
||||
"""The old writer stopped the field at the first '*' and swallowed the rest."""
|
||||
text = 'העמדה כוללת **הדגשה** וגם קו --- באמצע\nושורה שנייה.'
|
||||
research_md.update_chair_position(analysis_file, "issue_1", text)
|
||||
|
||||
assert _positions(analysis_file)["issue_1"] == text
|
||||
|
||||
|
||||
def test_update_does_not_touch_sibling_subsections(analysis_file: Path) -> None:
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", "ראשונה")
|
||||
research_md.update_chair_position(analysis_file, "threshold_2", "שנייה")
|
||||
|
||||
positions = _positions(analysis_file)
|
||||
assert positions["threshold_1"] == "ראשונה"
|
||||
assert positions["threshold_2"] == "שנייה"
|
||||
assert positions["issue_1"] == ""
|
||||
|
||||
|
||||
def test_closing_rule_and_bullet_marker_are_preserved(analysis_file: Path) -> None:
|
||||
"""An update must not restructure the file the analyst produced."""
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", "עמדה")
|
||||
research_md.update_chair_position(analysis_file, "issue_1", "עמדה")
|
||||
content = analysis_file.read_text(encoding="utf-8")
|
||||
|
||||
assert "- **עמדת ועדת הערר:**" in content
|
||||
assert content.rstrip().endswith("---")
|
||||
|
||||
|
||||
def test_clearing_a_position_restores_the_placeholder(analysis_file: Path) -> None:
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", "עמדה")
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", "")
|
||||
|
||||
assert _positions(analysis_file)["threshold_1"] == ""
|
||||
assert research_md.CHAIR_POSITION_PLACEHOLDERS[0] in analysis_file.read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def test_repeated_saves_do_not_accumulate_blocks(analysis_file: Path) -> None:
|
||||
for text in ("ראשון", "שני", "שלישי"):
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", text)
|
||||
|
||||
content = analysis_file.read_text(encoding="utf-8")
|
||||
assert content.count(f"**{research_md.CHAIR_POSITION_LABEL}:**") == 3
|
||||
assert _positions(analysis_file)["threshold_1"] == "שלישי"
|
||||
|
||||
|
||||
# ── decorated and duplicated labels ─────────────────────────────
|
||||
|
||||
|
||||
def test_decorated_label_is_updated_not_duplicated(tmp_path: Path) -> None:
|
||||
"""'עמדת ועדת הערר (הכוונת יו"ר 24.6)' appears in the corpus (8125-09-24)."""
|
||||
path = tmp_path / "a.md"
|
||||
path.write_text(
|
||||
"## 6. סוגיות להכרעה\n\n"
|
||||
"### סוגיה 1: כותרת\n\n"
|
||||
'**עמדת ועדת הערר (הכוונת יו"ר 24.6):** [ימולא ע"י יו"ר הוועדה]\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
research_md.update_chair_position(path, "issue_1", "עמדה מעודכנת")
|
||||
|
||||
content = path.read_text(encoding="utf-8")
|
||||
assert content.count(research_md.CHAIR_POSITION_LABEL) == 1
|
||||
assert _positions(path)["issue_1"] == "עמדה מעודכנת"
|
||||
|
||||
|
||||
def test_duplicate_labels_read_back_the_one_that_was_written(tmp_path: Path) -> None:
|
||||
"""An H3 covering several issues (1033-02-25) must not read a sibling's text."""
|
||||
path = tmp_path / "a.md"
|
||||
path.write_text(
|
||||
"## 6. סוגיות להכרעה\n\n"
|
||||
"### סוגיות 4–6: מקובצות\n\n"
|
||||
"**עמדת ועדת הערר:** ראשונה\n\n"
|
||||
"**עמדת ועדת הערר:** אחרונה\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
research_md.update_chair_position(path, "issue_1", "העמדה הנכונה")
|
||||
|
||||
assert _positions(path)["issue_1"] == "העמדה הנכונה"
|
||||
|
||||
|
||||
# ── read-after-write guard ──────────────────────────────────────
|
||||
|
||||
|
||||
def test_update_returns_the_persisted_value(analysis_file: Path) -> None:
|
||||
result = research_md.update_chair_position(analysis_file, "threshold_1", " עמדה ")
|
||||
|
||||
assert result["saved"] is True
|
||||
assert result["position"] == "עמדה"
|
||||
|
||||
|
||||
def test_a_write_the_parser_cannot_read_back_is_reported_as_failure(
|
||||
analysis_file: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Regression guard: the bug hid behind a success response for months."""
|
||||
monkeypatch.setattr(
|
||||
research_md, "_stored_chair_position", lambda *_args: "משהו אחר"
|
||||
)
|
||||
|
||||
with pytest.raises(RuntimeError, match="לא נקראה בחזרה"):
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", "עמדה")
|
||||
|
||||
|
||||
# ── downstream consumers see the recovered positions ────────────
|
||||
|
||||
|
||||
def test_chair_directions_count_bulleted_positions(analysis_file: Path) -> None:
|
||||
"""extract_chair_directions feeds legal-writer; it under-reported before."""
|
||||
research_md.update_chair_position(analysis_file, "threshold_1", "עמדה")
|
||||
|
||||
directions = research_md.extract_chair_directions(analysis_file)
|
||||
assert directions["filled_count"] == 1
|
||||
assert directions["status"] == "partial"
|
||||
filled = [t for t in directions["threshold_claims"] if t["direction"]]
|
||||
assert [t["id"] for t in filled] == ["threshold_1"]
|
||||
|
||||
|
||||
# ── the real corpus ─────────────────────────────────────────────
|
||||
|
||||
|
||||
CORPUS = sorted(
|
||||
Path("/home/chaim/legal-ai/data/cases").glob(
|
||||
"*/documents/research/analysis-and-research.md"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not CORPUS, reason="case corpus not present on this host")
|
||||
@pytest.mark.parametrize("source", CORPUS, ids=lambda p: p.parts[-4])
|
||||
def test_every_corpus_subsection_round_trips(source: Path, tmp_path: Path) -> None:
|
||||
sentinel = "עמדת-בדיקה — **הדגשה** ו---קו\nושורה שנייה."
|
||||
before = _positions(source)
|
||||
|
||||
for section_id in before:
|
||||
working = tmp_path / f"{section_id}.md"
|
||||
working.write_text(source.read_text(encoding="utf-8"), encoding="utf-8")
|
||||
|
||||
research_md.update_chair_position(working, section_id, sentinel)
|
||||
after = _positions(working)
|
||||
|
||||
assert after[section_id] == sentinel, f"{source.parts[-4]} {section_id}"
|
||||
untouched = {k: v for k, v in after.items() if k != section_id}
|
||||
assert untouched == {k: v for k, v in before.items() if k != section_id}
|
||||
@@ -166,17 +166,33 @@ def aggregate_cell(per_run: list[dict]) -> dict:
|
||||
"""Mean each metric across repeated generations of the same (case, block, effort)."""
|
||||
if not per_run:
|
||||
return {"distance": 1.0, "anti_pattern_total": 0.0, "change_percent": 100.0,
|
||||
"golden_ratio_deviation_pp": None, "n": 0}
|
||||
"golden_ratio_deviation_pp": None, "anti_by_pattern": {}, "n": 0}
|
||||
ratios = [r["golden_ratio_deviation_pp"] for r in per_run if r.get("golden_ratio_deviation_pp") is not None]
|
||||
return {
|
||||
"distance": round(mean(r["distance"] for r in per_run), 4),
|
||||
"anti_pattern_total": round(mean(r["anti_pattern_total"] for r in per_run), 2),
|
||||
"change_percent": round(mean(r["change_percent"] for r in per_run), 2),
|
||||
"golden_ratio_deviation_pp": round(mean(ratios), 2) if ratios else None,
|
||||
"anti_by_pattern": _mean_by_pattern(per_run),
|
||||
"n": len(per_run),
|
||||
}
|
||||
|
||||
|
||||
def _mean_by_pattern(per_run: list[dict]) -> dict:
|
||||
"""Mean hits PER anti-pattern name across runs — the 'which rule broke' view.
|
||||
|
||||
A pattern absent from a run counts as 0 (count_anti_patterns omits zero-hit
|
||||
keys), so the mean is over ALL runs, not only the ones that tripped it.
|
||||
"""
|
||||
names: set[str] = set()
|
||||
for r in per_run:
|
||||
names |= set((r.get("anti_by_pattern") or {}).keys())
|
||||
return {
|
||||
name: round(mean((r.get("anti_by_pattern") or {}).get(name, 0) for r in per_run), 2)
|
||||
for name in sorted(names)
|
||||
}
|
||||
|
||||
|
||||
def _current_default(block_id: str) -> str | None:
|
||||
from legal_mcp.services.block_writer import BLOCK_CONFIG, DEFAULT_EFFORT
|
||||
cfg = BLOCK_CONFIG.get(block_id, {})
|
||||
@@ -420,13 +436,30 @@ async def _finals_for_calibration(case_filter: str | None) -> list[dict]:
|
||||
|
||||
|
||||
async def _score_cell(case_id, block_id: str, effort: str, final_section: str,
|
||||
final_total_words: int, outcome: str, repeats: int) -> dict:
|
||||
"""Generate `block_id` at `effort` `repeats` times; score each vs the final section."""
|
||||
final_total_words: int, outcome: str, repeats: int,
|
||||
model: str | None = None, instructions: str = "") -> dict:
|
||||
"""Generate `block_id` at `effort` `repeats` times; score each vs the final section.
|
||||
|
||||
`model` (optional) A/Bs the generation model via write_block(model_override=…).
|
||||
None ⇒ the pinned GENERATION_MODEL, i.e. the production path unchanged.
|
||||
|
||||
`instructions` (optional) is appended to the block prompt for EVERY cell in
|
||||
the run — a prompt-variant A/B (e.g. an explicit formatting rule). It is
|
||||
applied to all models so the comparison stays a model comparison rather
|
||||
than silently becoming a prompt comparison.
|
||||
"""
|
||||
from legal_mcp.services import block_writer
|
||||
from legal_mcp.services.style_distance import block_distance_to_final
|
||||
runs: list[dict] = []
|
||||
models_used: list[str] = []
|
||||
for _ in range(repeats):
|
||||
res = await block_writer.write_block(case_id, block_id, effort_override=effort)
|
||||
res = await block_writer.write_block(
|
||||
case_id, block_id, instructions=instructions,
|
||||
effort_override=effort, model_override=model,
|
||||
)
|
||||
# Record what the CLI was actually asked to run, so a silent fallback to
|
||||
# a different build is visible in the report rather than mis-attributed.
|
||||
models_used.append(res.get("model_used") or "?")
|
||||
scored = block_distance_to_final(
|
||||
block_id, res.get("content", ""), final_section, outcome,
|
||||
section_target_total_words=final_total_words,
|
||||
@@ -434,6 +467,8 @@ async def _score_cell(case_id, block_id: str, effort: str, final_section: str,
|
||||
runs.append(scored)
|
||||
agg = aggregate_cell(runs)
|
||||
agg["effort"] = effort
|
||||
agg["model"] = model
|
||||
agg["models_used"] = sorted(set(models_used))
|
||||
agg["runs"] = runs
|
||||
return agg
|
||||
|
||||
@@ -446,6 +481,7 @@ async def _run(args, ts: str) -> dict:
|
||||
|
||||
efforts = args.efforts
|
||||
blocks = args.blocks
|
||||
models = args.models
|
||||
finals = await _finals_for_calibration(args.case)
|
||||
|
||||
cases_meta = []
|
||||
@@ -468,11 +504,15 @@ async def _run(args, ts: str) -> dict:
|
||||
section = _BLOCK_TO_SECTION.get(block_id)
|
||||
plan[block_id] = [c for c in cases_meta if section and c["sections"].get(section)]
|
||||
|
||||
total_cells = sum(len(plan[b]) for b in blocks) * len(efforts) * args.repeats
|
||||
total_cells = sum(len(plan[b]) for b in blocks) * len(efforts) * args.repeats * len(models)
|
||||
grid_summary = {
|
||||
"n_finals": len(cases_meta),
|
||||
"finals": [c["case_number"] for c in cases_meta],
|
||||
"blocks": blocks, "efforts": efforts, "repeats": args.repeats,
|
||||
"models": models,
|
||||
# Provenance: a prompt-variant run is NOT comparable to a baseline run,
|
||||
# so the instruction text is recorded in the report, not just the shell.
|
||||
"instructions": getattr(args, "instructions", "") or "",
|
||||
"total_generations": total_cells,
|
||||
"per_block_n": {b: len(plan[b]) for b in blocks},
|
||||
}
|
||||
@@ -480,6 +520,27 @@ async def _run(args, ts: str) -> dict:
|
||||
if args.dry_run:
|
||||
return {"dry_run": True, "grid": grid_summary, "by_block": {}}
|
||||
|
||||
by_model: dict[str, dict] = {}
|
||||
for model in models:
|
||||
by_block = await _run_blocks_for_model(
|
||||
model, blocks, efforts, plan, args, ts, grid_summary, by_model, _BLOCK_TO_SECTION,
|
||||
)
|
||||
by_model[model] = by_block
|
||||
|
||||
# `by_block` stays the single-model shape (first model) so --rerank and the
|
||||
# existing per-block report path keep working unchanged (G2 — no second
|
||||
# result schema); multi-model runs additionally carry by_model.
|
||||
out = {"dry_run": False, "grid": grid_summary, "by_block": by_model[models[0]]}
|
||||
if len(models) > 1:
|
||||
out["by_model"] = by_model
|
||||
return out
|
||||
|
||||
|
||||
async def _run_blocks_for_model(model, blocks, efforts, plan, args, ts, grid_summary,
|
||||
by_model_so_far, _BLOCK_TO_SECTION) -> dict:
|
||||
"""The per-block × per-effort grid for ONE generation model."""
|
||||
from uuid import UUID
|
||||
|
||||
by_block: dict[str, dict] = {}
|
||||
for block_id in blocks:
|
||||
section = _BLOCK_TO_SECTION.get(block_id)
|
||||
@@ -497,11 +558,12 @@ async def _run(args, ts: str) -> dict:
|
||||
cell = await _score_cell(
|
||||
UUID(c["case_id"]), block_id, effort, final_section,
|
||||
c["final_total_words"], c["outcome"], args.repeats,
|
||||
model=model, instructions=getattr(args, "instructions", "") or "",
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 — harness must survive any cell failure
|
||||
logger.warning(
|
||||
"calibration cell skipped: case=%s block=%s effort=%s — %s",
|
||||
c["case_number"], block_id, effort, exc,
|
||||
"calibration cell skipped: case=%s block=%s effort=%s model=%s — %s",
|
||||
c["case_number"], block_id, effort, model, exc,
|
||||
)
|
||||
continue
|
||||
per_effort_runs[effort].append(cell)
|
||||
@@ -523,6 +585,7 @@ async def _run(args, ts: str) -> dict:
|
||||
"anti_pattern_total": round(mean(r["anti_pattern_total"] for r in rows), 2),
|
||||
"change_percent": round(mean(r["change_percent"] for r in rows), 2),
|
||||
"golden_ratio_deviation_pp": round(mean(ratios), 2) if ratios else None,
|
||||
"anti_by_pattern": _mean_by_pattern(rows),
|
||||
"n": len(rows),
|
||||
})
|
||||
rec = recommend_effort(effort_rows)
|
||||
@@ -532,6 +595,11 @@ async def _run(args, ts: str) -> dict:
|
||||
"recommended": rec["effort"] if rec else None,
|
||||
"confidence": rec["confidence"] if rec else None,
|
||||
"confidence_margin": rec.get("confidence_margin") if rec else None,
|
||||
"model": model,
|
||||
# Model builds the CLI actually reported across this block's cells —
|
||||
# a mismatch vs `model` means a silent fallback, not a real A/B.
|
||||
"models_used": sorted({m for e in per_effort_runs.values()
|
||||
for cell in e for m in cell.get("models_used", [])}),
|
||||
"efforts": effort_rows,
|
||||
"per_case": per_case,
|
||||
}
|
||||
@@ -541,11 +609,15 @@ async def _run(args, ts: str) -> dict:
|
||||
# Blocks not yet done are simply absent from by_block; _write_report tolerates
|
||||
# partial results. main() does the final flush once the loop finishes.
|
||||
try:
|
||||
_write_report({"dry_run": False, "grid": grid_summary, "by_block": by_block}, ts)
|
||||
snap = {"dry_run": False, "grid": grid_summary, "by_block": by_block}
|
||||
if by_model_so_far or len(grid_summary.get("models", [])) > 1:
|
||||
snap["by_model"] = {**by_model_so_far, model: by_block}
|
||||
_write_report(snap, ts)
|
||||
except Exception as exc: # noqa: BLE001 — a write hiccup must not abort the run
|
||||
logger.warning("incremental report write failed after block=%s — %s", block_id, exc)
|
||||
logger.warning("incremental report write failed after block=%s model=%s — %s",
|
||||
block_id, model, exc)
|
||||
|
||||
return {"dry_run": False, "grid": grid_summary, "by_block": by_block}
|
||||
return by_block
|
||||
|
||||
|
||||
IL_TZ = ZoneInfo("Asia/Jerusalem")
|
||||
@@ -578,7 +650,10 @@ def _write_report(result: dict, ts: str) -> tuple[Path, Path]:
|
||||
"ההמלצה אדוויזורית; ההכרעה בידי היו\"ר/המפעיל.\n",
|
||||
f"- בלוקים: {', '.join(g['blocks'])}",
|
||||
f"- efforts: {', '.join(g['efforts'])} · repeats/cell: {g['repeats']}",
|
||||
f"- models: {', '.join(m or 'pinned-default' for m in g.get('models', [None]))}",
|
||||
f"- סך ייצורי-מודל: {g['total_generations']}",
|
||||
(f"- ⚠️ **וריאנט-פרומפט** (לא בר-השוואה לריצת-בסיס): `{g['instructions']}`"
|
||||
if g.get("instructions") else "- וריאנט-פרומפט: — (פרומפט ייצור כפי-שהוא)"),
|
||||
"",
|
||||
]
|
||||
if result.get("dry_run"):
|
||||
@@ -613,6 +688,54 @@ def _write_report(result: dict, ts: str) -> tuple[Path, Path]:
|
||||
f"| {r['effort']}{star} | {r['distance']:.4f} | {r['anti_pattern_total']} | "
|
||||
f"{r['change_percent']} | {ratio if ratio is not None else '—'} | {r['n']} |")
|
||||
lines.append("")
|
||||
by_model = result.get("by_model") or {}
|
||||
if len(by_model) > 1:
|
||||
lines += ["## השוואת-מודלים (אותו block, אותו effort, אותם סופיים)\n",
|
||||
"| block | effort | model | anti_total | change% | ratioΔpp | distance | n |",
|
||||
"|---|---|---|---|---|---|---|---|"]
|
||||
for b in g["blocks"]:
|
||||
for eff in g["efforts"]:
|
||||
rows = []
|
||||
for m, bb in by_model.items():
|
||||
for r in (bb.get(b) or {}).get("efforts", []):
|
||||
if r["effort"] == eff:
|
||||
rows.append((m, r))
|
||||
if len(rows) < 2:
|
||||
continue # nothing to compare for this cell — don't fake a row
|
||||
best = min(rows, key=lambda mr: (mr[1]["anti_pattern_total"],
|
||||
mr[1]["golden_ratio_deviation_pp"] or 0,
|
||||
mr[1]["distance"]))[0]
|
||||
for m, r in rows:
|
||||
ratio = r["golden_ratio_deviation_pp"]
|
||||
star = " ⭐" if m == best else ""
|
||||
lines.append(
|
||||
f"| {b} | {eff} | {m}{star} | {r['anti_pattern_total']} | "
|
||||
f"{r['change_percent']} | {ratio if ratio is not None else '—'} | "
|
||||
f"{r['distance']:.4f} | {r['n']} |")
|
||||
lines.append("")
|
||||
|
||||
# WHICH rule broke — a total alone can't tell you what to fix.
|
||||
bd_rows = [(b, eff, m, r) for b in g["blocks"] for eff in g["efforts"]
|
||||
for m, bb in by_model.items()
|
||||
for r in (bb.get(b) or {}).get("efforts", []) if r["effort"] == eff]
|
||||
if any(r.get("anti_by_pattern") for *_, r in bd_rows):
|
||||
names = sorted({n for *_, r in bd_rows for n in (r.get("anti_by_pattern") or {})})
|
||||
lines += ["### פילוח אנטי-דפוסים (איזה כלל הופר)\n",
|
||||
"| block | effort | model | " + " | ".join(names) + " |",
|
||||
"|---|---|---|" + "---|" * len(names)]
|
||||
for b, eff, m, r in bd_rows:
|
||||
cells = " | ".join(str((r.get("anti_by_pattern") or {}).get(n, 0)) for n in names)
|
||||
lines.append(f"| {b} | {eff} | {m} | {cells} |")
|
||||
lines.append("")
|
||||
# A silent CLI fallback would make the whole comparison meaningless — surface it.
|
||||
for m, bb in by_model.items():
|
||||
for b, bd in bb.items():
|
||||
used = bd.get("models_used") or []
|
||||
if used and any(not u.startswith(str(m)) for u in used):
|
||||
lines.append(f"> ⚠️ **{b} / {m}**: ה-CLI דיווח `{', '.join(used)}` — "
|
||||
"ייתכן fallback שקט; ההשוואה לתא זה אינה תקפה.\n")
|
||||
lines.append("")
|
||||
|
||||
lines.append("> דירוג-ההמלצה **style-clean** (#213): anti_total ראשי → ratioΔ → distance (tiebreak). "
|
||||
"**change% מדווח-לא-מדורג** — מערבב סגנון עם שלמות-תוכן (07-learning §0.7), "
|
||||
"anti_total הוא הסיגנל הנקי-לסגנון. confidence=⚠️weak ⇒ הבחירה בתוך-הרעש "
|
||||
@@ -633,6 +756,12 @@ async def main() -> int:
|
||||
help="comma block ids to calibrate")
|
||||
ap.add_argument("--case", default=None, help="restrict to a single case_number")
|
||||
ap.add_argument("--repeats", type=int, default=1, help="generations per cell (avg out gen noise)")
|
||||
ap.add_argument("--models", default="",
|
||||
help="comma generation-model ids to A/B (e.g. claude-opus-4-8,claude-opus-5). "
|
||||
"Empty (default) = the pinned GENERATION_MODEL, i.e. production unchanged.")
|
||||
ap.add_argument("--instructions", default="",
|
||||
help="extra prompt instruction appended to EVERY cell (prompt-variant A/B). "
|
||||
"Applied to all models — the run stays a model comparison. Recorded in the report.")
|
||||
args = ap.parse_args()
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
@@ -653,6 +782,9 @@ async def main() -> int:
|
||||
if bad_b:
|
||||
print(f"non-calibratable block(s): {bad_b}. valid: {VALID_BLOCKS}", file=sys.stderr)
|
||||
return 2
|
||||
# [None] = "use the pinned GENERATION_MODEL" — keeps the default run byte-identical
|
||||
# to the pre-#models behaviour instead of hard-coding the id in a second place (G2).
|
||||
args.models = [m.strip() for m in args.models.split(",") if m.strip()] or [None]
|
||||
|
||||
ts = _ts()
|
||||
result = await _run(args, ts)
|
||||
|
||||
@@ -45,8 +45,10 @@ export function ChairEditor({
|
||||
if (trimmed === lastSaved.current.trim()) return;
|
||||
setState({ kind: "saving" });
|
||||
try {
|
||||
await mutate.mutateAsync({ sectionId, position: trimmed });
|
||||
lastSaved.current = trimmed;
|
||||
const res = await mutate.mutateAsync({ sectionId, position: trimmed });
|
||||
/* Track what the backend read back, not what we sent — "✓ נשמר" must
|
||||
mean "persisted and re-readable", or a later blur skips the save. */
|
||||
lastSaved.current = res?.position ?? trimmed;
|
||||
setState({ kind: "saved", at: new Date() });
|
||||
} catch (e) {
|
||||
setState({
|
||||
|
||||
@@ -59,20 +59,31 @@ export function useResearchAnalysis(caseNumber: string | undefined) {
|
||||
});
|
||||
}
|
||||
|
||||
export type SaveChairPositionResult = {
|
||||
saved: boolean;
|
||||
section_id: string;
|
||||
/** What the backend read back out of the file after writing — the truth. */
|
||||
position: string;
|
||||
timestamp?: string;
|
||||
};
|
||||
|
||||
export function useSaveChairPosition(caseNumber: string | undefined) {
|
||||
const qc = useQueryClient();
|
||||
return useMutation({
|
||||
mutationFn: async (vars: { sectionId: string; position: string }) =>
|
||||
apiRequest<unknown>(
|
||||
apiRequest<SaveChairPositionResult>(
|
||||
`/api/cases/${caseNumber}/research/analysis/chair-position`,
|
||||
{
|
||||
method: "PATCH",
|
||||
body: { section_id: vars.sectionId, position: vars.position },
|
||||
},
|
||||
),
|
||||
onSuccess: (_res, vars) => {
|
||||
onSuccess: (res, vars) => {
|
||||
/* Locally patch the cached analysis so other consumers stay in sync
|
||||
without an immediate refetch that would steal focus from the editor. */
|
||||
without an immediate refetch that would steal focus from the editor.
|
||||
Cache the value the server read back, never the value we sent — the
|
||||
two diverged silently while the parser could not see bulleted fields. */
|
||||
const persisted = res?.position ?? vars.position;
|
||||
qc.setQueryData<ResearchAnalysis | undefined>(
|
||||
researchKeys.analysis(caseNumber ?? ""),
|
||||
(prev) => {
|
||||
@@ -80,7 +91,7 @@ export function useSaveChairPosition(caseNumber: string | undefined) {
|
||||
const patch = (arr?: ResearchSubsection[]) =>
|
||||
arr?.map((s) =>
|
||||
s.id === vars.sectionId
|
||||
? { ...s, chair_position: vars.position }
|
||||
? { ...s, chair_position: persisted }
|
||||
: s,
|
||||
);
|
||||
return {
|
||||
|
||||
Reference in New Issue
Block a user