feat(rag): Stage B — RAG improvements (HNSW + BM25 hybrid + MMR + dynamic boost)
All checks were successful
Build & Deploy / build-and-deploy (push) Successful in 1m35s

Five enhancements to the precedent retrieval stack:

* **#44 HNSW indexes** for precedent_chunks + halachot (replacing IVFFlat
  lists=50). Build time ~3s combined. Better recall@10 with pgvector 0.8.2.
* **#45 Halacha sweep** — 96 pending halachot at conf>=0.78 promoted to
  approved (1141 → 1237). Cluster at conf=0.78 spot-checked OK. Applied
  via psql only — env HALACHA_AUTO_APPROVE_THRESHOLD unchanged (0.80).
* **#43 MMR diversity** — search_precedent_library_hybrid now caps at
  ``max_per_case_law=2`` (default). Prevents one precedent dominating
  top-10 when many of its chunks/halachot rank high. New helper
  ``_diversify_by_case_law`` in hybrid_search.py.
* **#46 Dynamic halacha boost** — replaces the static ``score+=0.05``
  with ``score+=confidence*0.06``. Calibrated so avg-confidence (~0.85)
  stays at +0.05; high-conf halachot get a slight extra lift, low-conf
  ones get less. Behaviour preserved at the mean.
* **#41 BM25/tsvector hybrid + RRF**. Schema V12 adds STORED tsvector
  columns ``precedent_chunks.content_tsv`` and ``halachot.rule_tsv``
  (using simple config — Postgres has no Hebrew stemmer) + GIN indexes.
  New ``db.search_precedent_library_lexical`` mirrors the semantic
  function with ts_rank_cd over plainto_tsquery. ``hybrid_search``
  runs sem+lex in parallel and fuses via RRF before rerank. Toggle:
  env ``BM25_HYBRID_ENABLED`` (default true), graceful fallback to
  semantic-only on lexical failure.

#40 (VOYAGE_RERANK_ENABLED) was already true in Coolify env; no change.
#42 (Claude Haiku query expansion) deferred — latency + cost concerns
warrant a separate plan; the bm25 lexical leg already recovers most of
the exact-string recall #42 was meant to address.

Closes TaskMaster #41, #43-#46.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-26 08:08:02 +00:00
parent b197d2329c
commit af651d0135
3 changed files with 370 additions and 6 deletions

View File

@@ -87,6 +87,20 @@ MULTIMODAL_TEXT_WEIGHT = float(
# concentrate weight at top ranks; higher values flatten the curve.
MULTIMODAL_RRF_K = int(os.environ.get("MULTIMODAL_RRF_K", "60"))
# BM25/lexical hybrid — fuse ``ts_rank_cd`` over ``content_tsv``/
# ``rule_tsv`` (DB schema V12) with the semantic cosine layer via RRF.
# Recovers recall on exact-string queries that voyage embeddings blur
# (e.g. case-number citations like "1461/20", "317/10"; rare planning
# vocabulary). Hebrew uses the ``simple`` text-search config — no
# stemmer needed, and numeric/punctuation tokens stay intact. When
# disabled, hybrid search falls back to semantic-only (the previous
# behaviour). On by default — the lexical leg is cheap (GIN index) and
# only ever *adds* candidates to RRF, it can't down-rank a strong
# semantic hit.
BM25_HYBRID_ENABLED = (
os.environ.get("BM25_HYBRID_ENABLED", "true").lower() == "true"
)
# Halacha extraction — auto-approve threshold. Halachot with extractor
# confidence >= this value are inserted with review_status='approved'
# instead of 'pending_review' (so they immediately appear in