feat(storage): X14 Phase 1 — unified storage layer (services/storage.py)
The single choke-point for all binary file I/O (originals, derived artifacts, exports), replacing the scattered open()/shutil/Path.write_bytes calls across ~8 services. Backend chosen by STORAGE_BACKEND: - filesystem (default): disk under DATA_DIR — byte-for-byte legacy behaviour - dual: write disk + S3, read S3→disk fallback (migration window) - s3: MinIO via aioboto3 (lazy import; absent in the filesystem path) Keys are DATA_DIR-relative POSIX paths; the FS backend ignores the logical bucket and keeps the existing single tree, so the default backend is zero behaviour change. S3 maps a governance bucket (documents/immutable/derived) → MinIO bucket; presigned URLs are minted against the public endpoint (browser-reachable) and carry the Hebrew filename via RFC-5987 Content-Disposition. - config: STORAGE_BACKEND + MINIO_* (endpoint, public-endpoint, creds, region, 3 bucket names, presign TTL) - mcp_env_catalog: new "storage" category + 10 specs (X10/INV-ENV1) - pyproject: aioboto3>=13 (consumed here, deployed with first use) - tests: 18 unit tests (FS round-trip, key normalization/traversal guard, bucket resolution, backend selection, dual write-both + S3-down fallback) No call-sites are rewired yet — that is Phase 2 (106.3). STORAGE_BACKEND stays filesystem in prod, so behaviour is unchanged. Invariants: keeps G2 (one storage path replaces scattered I/O); establishes INV-STG1 (single layer), INV-STG2 (atomic keys, Hebrew name in metadata), INV-STG3 (governance buckets), INV-STG6 (presigned serving). Spec: docs/spec/X14-storage-minio.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -13,7 +13,8 @@ from typing import Any, Literal
|
||||
|
||||
EnvType = Literal["bool", "int", "float", "string"]
|
||||
EnvCategory = Literal[
|
||||
"multimodal", "rerank", "halacha", "credentials", "connection", "general"
|
||||
"multimodal", "rerank", "halacha", "credentials", "connection",
|
||||
"storage", "general"
|
||||
]
|
||||
|
||||
|
||||
@@ -89,6 +90,58 @@ ENV_CATALOG: dict[str, EnvSpec] = {
|
||||
"סף confidence ל-auto-approve של הלכות שחולצו",
|
||||
is_secret=False, is_editable=True, default=0.80, min=0.0, max=1.0,
|
||||
),
|
||||
# ── storage (X14 / MinIO) ──────────────────────────────────────
|
||||
"STORAGE_BACKEND": EnvSpec(
|
||||
"STORAGE_BACKEND", "storage", "string",
|
||||
"מנוע אחסון: filesystem (דיסק) / dual (דיסק+S3) / s3 (MinIO בלבד)",
|
||||
is_secret=False, is_editable=True, default="filesystem",
|
||||
enum_values=("filesystem", "dual", "s3"),
|
||||
),
|
||||
"MINIO_ENDPOINT": EnvSpec(
|
||||
"MINIO_ENDPOINT", "storage", "string",
|
||||
"endpoint פנימי של MinIO (server-side, רשת Docker)",
|
||||
is_secret=False, is_editable=False, default="http://minio:9000",
|
||||
),
|
||||
"MINIO_PUBLIC_ENDPOINT": EnvSpec(
|
||||
"MINIO_PUBLIC_ENDPOINT", "storage", "string",
|
||||
"endpoint ציבורי ל-presigned URLs (גישת דפדפן)",
|
||||
is_secret=False, is_editable=False, default="https://s3.nautilus.marcusgroup.org",
|
||||
),
|
||||
"MINIO_ACCESS_KEY": EnvSpec(
|
||||
"MINIO_ACCESS_KEY", "storage", "string",
|
||||
"MinIO access key (service-account מוגבל ל-3 הדליות)",
|
||||
is_secret=True, is_editable=False,
|
||||
),
|
||||
"MINIO_SECRET_KEY": EnvSpec(
|
||||
"MINIO_SECRET_KEY", "storage", "string",
|
||||
"MinIO secret key",
|
||||
is_secret=True, is_editable=False,
|
||||
),
|
||||
"MINIO_REGION": EnvSpec(
|
||||
"MINIO_REGION", "storage", "string",
|
||||
"אזור S3 (MinIO מתעלם — לחתימת SigV4)",
|
||||
is_secret=False, is_editable=False, default="us-east-1",
|
||||
),
|
||||
"MINIO_BUCKET_DOCUMENTS": EnvSpec(
|
||||
"MINIO_BUCKET_DOCUMENTS", "storage", "string",
|
||||
"דלי מסמכי-מקור (versioning)",
|
||||
is_secret=False, is_editable=False, default="legal-documents",
|
||||
),
|
||||
"MINIO_BUCKET_IMMUTABLE": EnvSpec(
|
||||
"MINIO_BUCKET_IMMUTABLE", "storage", "string",
|
||||
"דלי החלטות סופיות (versioning + Object-Lock COMPLIANCE)",
|
||||
is_secret=False, is_editable=False, default="legal-immutable",
|
||||
),
|
||||
"MINIO_BUCKET_DERIVED": EnvSpec(
|
||||
"MINIO_BUCKET_DERIVED", "storage", "string",
|
||||
"דלי נגזרים (thumbnails / extracted — ניתן-לשחזור)",
|
||||
is_secret=False, is_editable=False, default="legal-derived",
|
||||
),
|
||||
"MINIO_PRESIGN_TTL": EnvSpec(
|
||||
"MINIO_PRESIGN_TTL", "storage", "int",
|
||||
"תוקף presigned URL בשניות (מקס' SigV4 = 7 ימים)",
|
||||
is_secret=False, is_editable=True, default=900, min=60, max=604800,
|
||||
),
|
||||
# ── general ────────────────────────────────────────────────────
|
||||
"VOYAGE_MODEL": EnvSpec(
|
||||
"VOYAGE_MODEL", "general", "string",
|
||||
|
||||
Reference in New Issue
Block a user