feat(calibration): style-clean recommend_effort (anti-primary) + confidence + --rerank (#213)

Make the effort-calibration recommendation style-clean (style acquisition is the goal — write like chair Dafna Tamir): - recommend_effort ranks by anti_pattern_total (PRIMARY, the clean style-rule violation count) → golden_ratio_deviation_pp (SECONDARY, structural-style, None=worst) → distance (faint final tiebreak). change_percent is EXCLUDED from the ranking (it mixes style with content-completeness per 07-learning §0.7 — noise for STYLE) and kept in the report as context only. - Add a confidence flag ("clear"/"weak"): a pick whose anti-pattern lead over the runner-up is within noise (margin < epsilon = max(0.5, 0.20·spread)) is flagged ⚠️ weak, so an over-claimed pick (the old confident block-zayin "low") is never shown as confident. Rendered as a confidence column in the report. - Add --rerank <report.json>: re-apply the new key to a saved calibration JSON offline (NO LLM, NO DB), print + write *-reranked.md. Re-ranks today's 8-final run without regenerating. - recommend_effort stays a single shared pure function reused by scripts/calibrate_block_yod.py (G2); both --self-test suites pass. - Spec note added (07-learning §0.7) + SCRIPTS.md updated. Invariants: G8 (style-clean empirical ranking), G2 (single shared recommend_effort, reused by the block-yod harness — no parallel metric path), INV-LRN5 (style-only signal; change% reported-not-ranked). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 19:00:41 +00:00
parent 4285288ba7
commit c7156b575e
3 changed files with 241 additions and 23 deletions
--- a/scripts/calibrate_effort.py
+++ b/scripts/calibrate_effort.py
@@ -19,9 +19,15 @@ WHAT IT MEASURES (per (case, block, effort) cell — the A/B grid):
    • distance                 — normalized composite (lower = closer to Dafna)
  No parallel metric path: it reuses style_distance + learning_loop (G2).

-RECOMMENDATION: for each block, the effort with the lowest MEAN composite distance
-across cases (ties → fewer anti-patterns → lower change_percent). Reported next to
-the #204 current default so a regression/improvement is visible.
+RECOMMENDATION (#213 — STYLE-CLEAN, anti-primary): for each block, the effort is
+ranked by `anti_pattern_total` FIRST (the clean Dafna-style-rule-violation count),
+then `golden_ratio_deviation_pp` (structural-style), and `distance` only as a faint
+final tiebreak. `change_percent` is REPORTED-NOT-RANKED: per 07-learning §0.7 it
+mixes style with content-completeness (a case's missing facts the model can't know)
+→ noise for STYLE. The pick carries a `confidence` flag ("clear"/"weak"): a pick
+whose anti-pattern lead over the runner-up is within noise is flagged "weak" so an
+over-claimed pick (the old block-zayin "low") is never shown as confident. Reported
+next to the #204 current default so a regression/improvement is visible.

 ⚠️ SAMPLE-SIZE CAVEAT (honored, not hidden): very few cases have an uploaded final
 (draft_final_pairs.final_text non-empty). The report prints n_finals PROMINENTLY and
@@ -94,20 +100,66 @@ VALID_EFFORTS = {"low", "medium", "high", "xhigh", "max"}


 # ── pure helpers (offline-testable) ──────────────────────────────────────────
-def recommend_effort(cells: list[dict]) -> dict | None:
-    """Pick the best effort for ONE block from its scored cells.
+# Style-clean ranking epsilon (#213). Confidence in a pick = how far the best
+# effort's anti_pattern_total beats the 2nd-best. We call a pick "clear" only when
+# that margin clears a floor that is itself the LARGER of:
+#   • an absolute floor (≥0.5 avg anti-patterns — a half-violation per case), and
+#   • a fraction of the spread across efforts (best can't claim a lead that is mere
+#     noise on a block whose efforts barely differ).
+# Below that → "weak" (within noise; nominal pick still reported, but flagged so a
+# within-noise pick is never shown as confident). Documented in the docstring.
+CONFIDENCE_ABS_FLOOR = 0.5       # avg anti-patterns (a half-violation per case)
+CONFIDENCE_SPREAD_FRACTION = 0.20  # of the across-effort anti spread

-    cells: [{"effort","distance","anti_pattern_total","change_percent","n"}].
-    Lowest mean composite distance wins; ties broken by fewer anti-patterns,
-    then lower change_percent. Pure → unit-tested in --self-test.
+
+def _anti_confidence(rows: list[dict]) -> tuple[str, float, float]:
+    """Confidence that the anti-primary pick is real, not within-noise.
+
+    Returns (confidence, margin, epsilon) where confidence ∈ {"clear","weak"}.
+    margin = (2nd-best anti) − (best anti); epsilon = the bar margin must clear.
+    A single effort (no rival) is trivially "clear". Pure → unit-tested.
+    """
+    if len(rows) < 2:
+        return "clear", float("inf"), 0.0
+    antis = sorted(r["anti_pattern_total"] for r in rows)
+    best, second = antis[0], antis[1]
+    margin = second - best
+    spread = antis[-1] - antis[0]
+    epsilon = max(CONFIDENCE_ABS_FLOOR, CONFIDENCE_SPREAD_FRACTION * spread)
+    return ("clear" if margin >= epsilon else "weak"), round(margin, 4), round(epsilon, 4)
+
+
+def recommend_effort(cells: list[dict]) -> dict | None:
+    """Pick the STYLE-CLEANEST effort for ONE block from its scored cells (#213).
+
+    cells: [{"effort","distance","anti_pattern_total","change_percent",
+             "golden_ratio_deviation_pp","n"}].
+
+    Ranking key (style-clean, G8 / 07-learning §0.7) — change_percent is EXCLUDED
+    from the ranking (it mixes style with content-completeness; reported only):
+      1. PRIMARY  anti_pattern_total           (↑ fewer Dafna-style violations = better)
+      2. SECONDARY golden_ratio_deviation_pp   (↑ structural-style; None → +inf worst)
+      3. TIEBREAK distance                     (↑ faint final tiebreak only)
+
+    Returns the winning row dict (so callers reading ["effort"] keep working) with
+    two added keys: ``confidence`` ∈ {"clear","weak"} and ``confidence_margin`` —
+    so a within-noise pick is never presented as confident. Pure → unit-tested.
    """
    if not cells:
        return None
    ranked = sorted(
        cells,
-        key=lambda c: (c["distance"], c["anti_pattern_total"], c["change_percent"]),
+        key=lambda c: (
+            c["anti_pattern_total"],
+            c["golden_ratio_deviation_pp"] if c.get("golden_ratio_deviation_pp") is not None else float("inf"),
+            c["distance"],
+        ),
    )
-    return ranked[0]
+    best = dict(ranked[0])
+    conf, margin, _eps = _anti_confidence(cells)
+    best["confidence"] = conf
+    best["confidence_margin"] = margin
+    return best


 def aggregate_cell(per_run: list[dict]) -> dict:
@@ -133,6 +185,88 @@ def _current_default(block_id: str) -> str | None:
    return cfg.get("effort", DEFAULT_EFFORT)


+# ── re-rank a saved report (offline — NO LLM, NO DB) ─────────────────────────
+def rerank_saved(saved: dict) -> dict:
+    """Re-apply the CURRENT (#213) recommend_effort to a saved calibration report.
+
+    Reads the per-(block,effort) rows that the run already persisted
+    (distance/anti/change%/ratioΔ/n) and recomputes recommended + confidence — no
+    regeneration, no model, no DB. Lets a past run be re-ranked under the new
+    style-clean key. Returns {"block_id": {old, new, confidence, confidence_margin,
+    current_default, efforts}}. Pure → unit-tested in --self-test.
+    """
+    out: dict[str, dict] = {}
+    for block_id, bd in (saved.get("by_block") or {}).items():
+        rows = bd.get("efforts") or []
+        rec = recommend_effort(rows)
+        out[block_id] = {
+            "section": bd.get("section"),
+            "current_default": bd.get("current_default"),
+            "old_recommended": bd.get("recommended"),
+            "new_recommended": rec["effort"] if rec else None,
+            "confidence": rec["confidence"] if rec else None,
+            "confidence_margin": rec.get("confidence_margin") if rec else None,
+            "efforts": rows,
+        }
+    return out
+
+
+def _render_rerank_md(saved: dict, reranked: dict, src: Path) -> str:
+    g = saved.get("grid", {})
+    finals = g.get("finals", [])
+    lines = [
+        f"# #213 — re-rank של {src.name} (style-clean, anti-primary · OFFLINE)\n",
+        f"> מקור: `{src.name}` · {g.get('n_finals','?')} סופיים "
+        f"({', '.join(finals) or '—'}) · ללא LLM/DB — דירוג-מחדש בלבד.\n",
+        "## old pick → new pick (+confidence)\n",
+        "| block | section | current | old pick | **new pick** | confidence | n |",
+        "|---|---|---|---|---|---|---|",
+    ]
+    for b, rd in reranked.items():
+        old = rd.get("old_recommended") or "—"
+        new = rd.get("new_recommended") or "—"
+        mark = "" if new == old else " ⬅︎"
+        conf = rd.get("confidence")
+        conf_cell = {"clear": "clear", "weak": "⚠️ weak (within-noise)"}.get(conf, "—")
+        n_b = rd["efforts"][0]["n"] if rd.get("efforts") else 0
+        lines.append(
+            f"| {b} | {rd.get('section') or '—'} | {rd.get('current_default') or '—'} | "
+            f"{old} | **{new}**{mark} | {conf_cell} | {n_b} |")
+    lines.append("")
+    for b, rd in reranked.items():
+        if not rd.get("efforts"):
+            continue
+        lines += [f"### {b} ({rd.get('section') or '—'})\n",
+                  "| effort | distance | anti_total | change% (reported) | ratioΔpp | n |",
+                  "|---|---|---|---|---|---|"]
+        for r in rd["efforts"]:
+            star = " ⭐" if r["effort"] == rd.get("new_recommended") else ""
+            ratio = r.get("golden_ratio_deviation_pp")
+            lines.append(
+                f"| {r['effort']}{star} | {r['distance']:.4f} | {r['anti_pattern_total']} | "
+                f"{r['change_percent']} | {ratio if ratio is not None else '—'} | {r['n']} |")
+        lines.append("")
+    lines.append("> דירוג **style-clean** (#213): anti_total ראשי → ratioΔ → distance (tiebreak); "
+                 "**change% מדווח-לא-מדורג** (07-learning §0.7); "
+                 "confidence=⚠️weak ⇒ הובלת-anti < epsilon = max(0.5, 0.20·spread).\n")
+    return "\n".join(lines) + "\n"
+
+
+def _do_rerank(path: Path) -> int:
+    """--rerank entry: load a saved report, re-rank offline, print + write *-reranked.md."""
+    if not path.exists():
+        print(f"rerank: file not found: {path}", file=sys.stderr)
+        return 2
+    saved = json.loads(path.read_text(encoding="utf-8"))
+    reranked = rerank_saved(saved)
+    md = _render_rerank_md(saved, reranked, path)
+    print(md)
+    out = path.with_name(path.stem + "-reranked.md")
+    out.write_text(md, encoding="utf-8")
+    print(f"  reranked report written: {out}", file=sys.stderr)
+    return 0
+
+
 # ── self-test (no DB, no model) ──────────────────────────────────────────────
 def _self_test() -> int:
    ok = True
@@ -180,16 +314,66 @@ def _self_test() -> int:
        {"background", "claims", "discussion", "summary"}))
    chk("split found ≥1 section", len(secs) >= 1)

-    # recommend_effort: lowest distance wins; tie → fewer anti-patterns.
+    # recommend_effort (#213, style-clean): anti_pattern_total is PRIMARY — the
+    # effort with the FEWEST style violations wins, even if its distance is higher.
    rec = recommend_effort([
-        {"effort": "low", "distance": 0.40, "anti_pattern_total": 5, "change_percent": 40, "n": 1},
-        {"effort": "high", "distance": 0.20, "anti_pattern_total": 2, "change_percent": 25, "n": 1},
-        {"effort": "xhigh", "distance": 0.20, "anti_pattern_total": 1, "change_percent": 22, "n": 1},
+        {"effort": "low", "distance": 0.20, "anti_pattern_total": 5, "change_percent": 22,
+         "golden_ratio_deviation_pp": 3.0, "n": 1},
+        {"effort": "high", "distance": 0.40, "anti_pattern_total": 2, "change_percent": 40,
+         "golden_ratio_deviation_pp": 5.0, "n": 1},
+        {"effort": "xhigh", "distance": 0.50, "anti_pattern_total": 1, "change_percent": 50,
+         "golden_ratio_deviation_pp": 9.0, "n": 1},
    ])
-    chk("recommend picks lowest distance", rec["distance"] == 0.20)
-    chk("recommend tie → fewer anti-patterns", rec["effort"] == "xhigh")
+    chk("recommend: anti-primary (not distance)", rec["effort"] == "xhigh")
+    chk("recommend: change_percent NOT ranked", rec["distance"] == 0.50)  # worst distance still won
    chk("recommend empty ⇒ None", recommend_effort([]) is None)

+    # SECONDARY = golden_ratio_deviation_pp when anti ties; distance only as last tiebreak.
+    rec_tie = recommend_effort([
+        {"effort": "low", "distance": 0.10, "anti_pattern_total": 3, "change_percent": 10,
+         "golden_ratio_deviation_pp": 8.0, "n": 1},
+        {"effort": "high", "distance": 0.90, "anti_pattern_total": 3, "change_percent": 90,
+         "golden_ratio_deviation_pp": 2.0, "n": 1},
+    ])
+    chk("recommend: ratio breaks anti-tie", rec_tie["effort"] == "high")
+    rec_ratio_tie = recommend_effort([
+        {"effort": "low", "distance": 0.30, "anti_pattern_total": 3, "change_percent": 10,
+         "golden_ratio_deviation_pp": 5.0, "n": 1},
+        {"effort": "high", "distance": 0.10, "anti_pattern_total": 3, "change_percent": 90,
+         "golden_ratio_deviation_pp": 5.0, "n": 1},
+    ])
+    chk("recommend: distance breaks final tie", rec_ratio_tie["effort"] == "high")
+    # None ratio is treated as worst (never beats a real deviation on the secondary key).
+    rec_none = recommend_effort([
+        {"effort": "low", "distance": 0.10, "anti_pattern_total": 2, "change_percent": 10,
+         "golden_ratio_deviation_pp": None, "n": 1},
+        {"effort": "high", "distance": 0.90, "anti_pattern_total": 2, "change_percent": 90,
+         "golden_ratio_deviation_pp": 4.0, "n": 1},
+    ])
+    chk("recommend: None ratio is worst", rec_none["effort"] == "high")
+
+    # confidence flag (#213): a clear anti lead ⇒ "clear"; within-noise ⇒ "weak".
+    rec_clear = recommend_effort([
+        {"effort": "low", "distance": 0.5, "anti_pattern_total": 1.0, "change_percent": 50,
+         "golden_ratio_deviation_pp": 5.0, "n": 5},
+        {"effort": "high", "distance": 0.5, "anti_pattern_total": 4.0, "change_percent": 50,
+         "golden_ratio_deviation_pp": 5.0, "n": 5},
+    ])
+    chk("confidence: big anti lead ⇒ clear", rec_clear["confidence"] == "clear")
+    rec_weak = recommend_effort([
+        {"effort": "low", "distance": 0.4, "anti_pattern_total": 3.6, "change_percent": 93,
+         "golden_ratio_deviation_pp": 6.7, "n": 5},
+        {"effort": "xhigh", "distance": 0.5, "anti_pattern_total": 3.6, "change_percent": 102,
+         "golden_ratio_deviation_pp": 6.6, "n": 5},
+        {"effort": "high", "distance": 0.53, "anti_pattern_total": 3.8, "change_percent": 104,
+         "golden_ratio_deviation_pp": 5.7, "n": 5},
+    ])
+    chk("confidence: anti tie ⇒ weak", rec_weak["confidence"] == "weak")
+    rec_single = recommend_effort([
+        {"effort": "low", "distance": 0.4, "anti_pattern_total": 2.0, "change_percent": 50,
+         "golden_ratio_deviation_pp": 5.0, "n": 5}])
+    chk("confidence: single effort ⇒ clear", rec_single["confidence"] == "clear")
+
    # aggregate_cell: means + ratio drops Nones, keeps n.
    agg = aggregate_cell([
        {"distance": 0.2, "anti_pattern_total": 2, "change_percent": 20, "golden_ratio_deviation_pp": 3.0},
@@ -199,6 +383,23 @@ def _self_test() -> int:
    chk("aggregate n counted", agg["n"] == 2)
    chk("aggregate ratio skips None", agg["golden_ratio_deviation_pp"] == 3.0)

+    # rerank_saved: re-applies the new key to a saved report's stored rows, no I/O.
+    # Mimics the real 8-final block-vav rows: low has fewest anti by a clear margin.
+    saved = {"grid": {"n_finals": 7, "finals": ["x"]}, "by_block": {
+        "block-vav": {"section": "background", "current_default": "medium",
+                      "recommended": "low", "efforts": [
+            {"effort": "low", "distance": 0.4454, "anti_pattern_total": 1.57, "change_percent": 87.39, "golden_ratio_deviation_pp": 6.33, "n": 7},
+            {"effort": "medium", "distance": 0.4749, "anti_pattern_total": 2.29, "change_percent": 87.76, "golden_ratio_deviation_pp": 6.37, "n": 7},
+            {"effort": "high", "distance": 0.5381, "anti_pattern_total": 4.14, "change_percent": 90.69, "golden_ratio_deviation_pp": 5.99, "n": 7},
+            {"effort": "xhigh", "distance": 0.5413, "anti_pattern_total": 4.57, "change_percent": 89.53, "golden_ratio_deviation_pp": 6.41, "n": 7},
+        ]},
+    }}
+    rr = rerank_saved(saved)
+    chk("rerank: block-vav stays low", rr["block-vav"]["new_recommended"] == "low")
+    chk("rerank: block-vav clear (anti lead)", rr["block-vav"]["confidence"] == "clear")
+    chk("rerank: keeps old pick for diff", rr["block-vav"]["old_recommended"] == "low")
+    chk("rerank: renders markdown", "new pick" in _render_rerank_md(saved, rr, Path("x.json")))
+
    print("ALL PASS" if ok else "*** FAILURES ***")
    return 0 if ok else 1

@@ -329,6 +530,8 @@ async def _run(args, ts: str) -> dict:
            "section": section,
            "current_default": _current_default(block_id),
            "recommended": rec["effort"] if rec else None,
+            "confidence": rec["confidence"] if rec else None,
+            "confidence_margin": rec.get("confidence_margin") if rec else None,
            "efforts": effort_rows,
            "per_case": per_case,
        }
@@ -386,13 +589,18 @@ def _write_report(result: dict, ts: str) -> tuple[Path, Path]:
        mp.write_text("\n".join(lines) + "\n", encoding="utf-8")
        return jp, mp

-    lines += ["## המלצה per-בלוק (distance נמוך = קרוב יותר לדפנה)\n",
-              "| block | section | current | **recommended** | n |", "|---|---|---|---|---|"]
+    lines += ["## המלצה per-בלוק (style-clean: anti_pattern_total ראשי — #213)\n",
+              "| block | section | current | **recommended** | confidence | n |",
+              "|---|---|---|---|---|---|"]
    for b, bd in result["by_block"].items():
        rec = bd.get("recommended") or "—"
        mark = "" if rec == bd.get("current_default") else " ⬅︎"
+        conf = bd.get("confidence")
+        conf_cell = {"clear": "clear", "weak": "⚠️ weak (within-noise)"}.get(conf, "—")
        n_b = bd["efforts"][0]["n"] if bd.get("efforts") else 0
-        lines.append(f"| {b} | {bd.get('section','')} | {bd.get('current_default') or '—'} | **{rec}**{mark} | {n_b} |")
+        lines.append(
+            f"| {b} | {bd.get('section','')} | {bd.get('current_default') or '—'} | "
+            f"**{rec}**{mark} | {conf_cell} | {n_b} |")
    lines.append("")
    for b, bd in result["by_block"].items():
        lines += [f"### {b} ({bd.get('section','')})\n",
@@ -405,8 +613,10 @@ def _write_report(result: dict, ts: str) -> tuple[Path, Path]:
                f"| {r['effort']}{star} | {r['distance']:.4f} | {r['anti_pattern_total']} | "
                f"{r['change_percent']} | {ratio if ratio is not None else '—'} | {r['n']} |")
        lines.append("")
-    lines.append("> change% מערבב סגנון עם שלמות-תוכן (07-learning §0.7); "
-                 "anti_total הוא הסיגנל הנקי-יותר לסגנון.\n")
+    lines.append("> דירוג-ההמלצה **style-clean** (#213): anti_total ראשי → ratioΔ → distance (tiebreak). "
+                 "**change% מדווח-לא-מדורג** — מערבב סגנון עם שלמות-תוכן (07-learning §0.7), "
+                 "anti_total הוא הסיגנל הנקי-לסגנון. confidence=⚠️weak ⇒ הבחירה בתוך-הרעש "
+                 "(ההובלה ב-anti קטנה מ-epsilon = max(0.5, 0.20·spread)).\n")
    mp.write_text("\n".join(lines) + "\n", encoding="utf-8")
    return jp, mp

@@ -414,6 +624,8 @@ def _write_report(result: dict, ts: str) -> tuple[Path, Path]:
 async def main() -> int:
    ap = argparse.ArgumentParser(description="#208 model/effort calibration harness")
    ap.add_argument("--self-test", action="store_true", help="offline measurement-logic proof (no DB/CLI)")
+    ap.add_argument("--rerank", metavar="REPORT.json", default=None,
+                    help="re-rank a saved calibration JSON under the current style-clean key (no LLM/DB)")
    ap.add_argument("--dry-run", action="store_true", help="plan the A/B grid over existing finals, no model calls")
    ap.add_argument("--efforts", default=",".join(DEFAULT_EFFORTS),
                    help=f"comma effort grid (default {','.join(DEFAULT_EFFORTS)})")
@@ -428,6 +640,9 @@ async def main() -> int:
    if args.self_test:
        return _self_test()

+    if args.rerank:
+        return _do_rerank(Path(args.rerank))
+
    args.efforts = [e.strip() for e in args.efforts.split(",") if e.strip()]
    bad = [e for e in args.efforts if e not in VALID_EFFORTS]
    if bad:
@@ -452,7 +667,8 @@ async def main() -> int:
    else:
        for b, bd in result["by_block"].items():
            print(f"  {b:16} current={bd.get('current_default') or '—':6} "
-                  f"→ recommended={bd.get('recommended') or '—'}")
+                  f"→ recommended={bd.get('recommended') or '—':6} "
+                  f"[{bd.get('confidence') or '—'}]")
    print(f"  report: {mp}")
    return 0