feat(storage): X14 Phase 1 — unified storage layer (services/storage.py)

The single choke-point for all binary file I/O (originals, derived
artifacts, exports), replacing the scattered open()/shutil/Path.write_bytes
calls across ~8 services. Backend chosen by STORAGE_BACKEND:
- filesystem (default): disk under DATA_DIR — byte-for-byte legacy behaviour
- dual: write disk + S3, read S3→disk fallback (migration window)
- s3: MinIO via aioboto3 (lazy import; absent in the filesystem path)

Keys are DATA_DIR-relative POSIX paths; the FS backend ignores the logical
bucket and keeps the existing single tree, so the default backend is zero
behaviour change. S3 maps a governance bucket (documents/immutable/derived)
→ MinIO bucket; presigned URLs are minted against the public endpoint
(browser-reachable) and carry the Hebrew filename via RFC-5987
Content-Disposition.

- config: STORAGE_BACKEND + MINIO_* (endpoint, public-endpoint, creds,
  region, 3 bucket names, presign TTL)
- mcp_env_catalog: new "storage" category + 10 specs (X10/INV-ENV1)
- pyproject: aioboto3>=13 (consumed here, deployed with first use)
- tests: 18 unit tests (FS round-trip, key normalization/traversal guard,
  bucket resolution, backend selection, dual write-both + S3-down fallback)

No call-sites are rewired yet — that is Phase 2 (106.3). STORAGE_BACKEND
stays filesystem in prod, so behaviour is unchanged.

Invariants: keeps G2 (one storage path replaces scattered I/O); establishes
INV-STG1 (single layer), INV-STG2 (atomic keys, Hebrew name in metadata),
INV-STG3 (governance buckets), INV-STG6 (presigned serving).
Spec: docs/spec/X14-storage-minio.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-08 07:47:49 +00:00
parent ade22ca871
commit b4a28f072d
5 changed files with 751 additions and 1 deletions

View File

@@ -13,7 +13,8 @@ from typing import Any, Literal
EnvType = Literal["bool", "int", "float", "string"]
EnvCategory = Literal[
"multimodal", "rerank", "halacha", "credentials", "connection", "general"
"multimodal", "rerank", "halacha", "credentials", "connection",
"storage", "general"
]
@@ -89,6 +90,58 @@ ENV_CATALOG: dict[str, EnvSpec] = {
"סף confidence ל-auto-approve של הלכות שחולצו",
is_secret=False, is_editable=True, default=0.80, min=0.0, max=1.0,
),
# ── storage (X14 / MinIO) ──────────────────────────────────────
"STORAGE_BACKEND": EnvSpec(
"STORAGE_BACKEND", "storage", "string",
"מנוע אחסון: filesystem (דיסק) / dual (דיסק+S3) / s3 (MinIO בלבד)",
is_secret=False, is_editable=True, default="filesystem",
enum_values=("filesystem", "dual", "s3"),
),
"MINIO_ENDPOINT": EnvSpec(
"MINIO_ENDPOINT", "storage", "string",
"endpoint פנימי של MinIO (server-side, רשת Docker)",
is_secret=False, is_editable=False, default="http://minio:9000",
),
"MINIO_PUBLIC_ENDPOINT": EnvSpec(
"MINIO_PUBLIC_ENDPOINT", "storage", "string",
"endpoint ציבורי ל-presigned URLs (גישת דפדפן)",
is_secret=False, is_editable=False, default="https://s3.nautilus.marcusgroup.org",
),
"MINIO_ACCESS_KEY": EnvSpec(
"MINIO_ACCESS_KEY", "storage", "string",
"MinIO access key (service-account מוגבל ל-3 הדליות)",
is_secret=True, is_editable=False,
),
"MINIO_SECRET_KEY": EnvSpec(
"MINIO_SECRET_KEY", "storage", "string",
"MinIO secret key",
is_secret=True, is_editable=False,
),
"MINIO_REGION": EnvSpec(
"MINIO_REGION", "storage", "string",
"אזור S3 (MinIO מתעלם — לחתימת SigV4)",
is_secret=False, is_editable=False, default="us-east-1",
),
"MINIO_BUCKET_DOCUMENTS": EnvSpec(
"MINIO_BUCKET_DOCUMENTS", "storage", "string",
"דלי מסמכי-מקור (versioning)",
is_secret=False, is_editable=False, default="legal-documents",
),
"MINIO_BUCKET_IMMUTABLE": EnvSpec(
"MINIO_BUCKET_IMMUTABLE", "storage", "string",
"דלי החלטות סופיות (versioning + Object-Lock COMPLIANCE)",
is_secret=False, is_editable=False, default="legal-immutable",
),
"MINIO_BUCKET_DERIVED": EnvSpec(
"MINIO_BUCKET_DERIVED", "storage", "string",
"דלי נגזרים (thumbnails / extracted — ניתן-לשחזור)",
is_secret=False, is_editable=False, default="legal-derived",
),
"MINIO_PRESIGN_TTL": EnvSpec(
"MINIO_PRESIGN_TTL", "storage", "int",
"תוקף presigned URL בשניות (מקס' SigV4 = 7 ימים)",
is_secret=False, is_editable=True, default=900, min=60, max=604800,
),
# ── general ────────────────────────────────────────────────────
"VOYAGE_MODEL": EnvSpec(
"VOYAGE_MODEL", "general", "string",