TranscriptX Storage Policy¶

This document defines the storage contract for path roots and the serialization rule. It is the reference for the path/storage architecture.

Persistence today is file-backed JSON plus FileLock. Live src/ has no SQLite, SQLAlchemy, or Alembic. A leftover data/state/transcriptx.db on disk is unused. A derived SQLite analytics layer is roadmap Theme J only (ROADMAP.md §J), not current architecture.

Serialization rule¶

Internally, filesystem paths are always represented as Path objects.

At configuration, CLI, JSON, or API boundaries, paths are serialized as strings.

Implications:

Internal code uses Path everywhere.
Config dataclasses keep str-typed fields for path values.
JSON serialization uses json.dump(..., default=str) or equivalent.
No Path objects should appear in serialized config files.

Configuration layer precedence and provenance labels¶

Effective configuration layer precedence is:
- Environment
- Run override (or Draft override when no run is selected)
- Project config
- Defaults
Current resolver provenance labels intentionally report draft overrides under the run-layer source model (source: run) to preserve existing semantics.
User-facing Settings scopes, Common vs Advanced knobs, and the overloaded “profile” taxonomy (install vs module/workflow vs analysis UI presets vs speaker profiles) are documented in settings.md.
Runtime module/workflow profile JSON lives under config_dir/profiles/ (override with TRANSCRIPTX_PROFILES_DIR). Tracked repo paths under data/profiles/*/default.json are allowlisted fixtures only — they are not auto-seeded into PROFILES_DIR, and ProfileManager treats disk name default as virtual (code defaults), not a loadable preset file.

Storage roots¶

Root	Meaning	Owner	Mountable	Safe to delete/rebuild	Authored by
recordings_dir	User media library (source audio)	user	yes	no (never auto-clean)	user
transcripts_dir	User transcript library	user	yes	no (never auto-clean)	user
data_dir	App working state, outputs, cache	app	optional	partially reconstructable	app
config_dir	User/app configuration	user/app	optional	no (not safe to auto-delete)	user/app
outputs_dir	Analysis run outputs	app	optional	yes (re-run to rebuild)	app
state_dir	Processing state, locks, rename journal	app	under data_dir	partially reconstructable	app
wav_backup_dir	WAV archive / reproducibility	user/app	optional	no (unless explicit)	app

Details¶

recordings_dir: User-owned, persistent, mountable, never auto-clean, user-authored content. Do not commit audio into the repository. With Docker Compose, set HOST_RECORDINGS_DIR to a host directory outside the clone; data/recordings/ under the repo is gitignored for local/native defaults only. Settings → Storage → Duplicate library files may delete extra copies after an explicit preview and typed confirmation; it is never automatic.
transcripts_dir: User-owned, persistent, mountable, never auto-clean, user-authored content. Settings → Storage → Duplicate library files may delete extra duplicate transcripts (and companions) after confirmation; Library may delete one selected transcript (and companions) after confirmation. Linked recordings and analysis run folders are left in place. Admission, rename, and analysis-run cleanup still never auto-delete this tree. Do not commit library JSON into the repository. Maintainer analysis probes live under tests/fixtures/ (see tests/fixtures/analysis_probes/README.md); deep-test and Docker one-shot runs must analyse those paths in place (TRANSCRIPTX_ALLOW_UNMANAGED_TRANSCRIPTS=1) and must not import them into this library.
- readable/ is a derived child within this library (not a peer root).
- imports/ is ephemeral staging owned by the managed import workflow.
- originals/ stores archived source files for managed imports (never overwritten; disambiguated names). Audio paths under originals/ (and any archival audio roots) are stable archival references and are intentionally decoupled from transcript filenames.
- metadata/ stores managed transcript sidecars under metadata-kind subtrees that mirror the transcript-relative path (for example, transcripts/foo/bar.json → metadata/imports/foo/bar.import_meta.json).
- Language variants (e.g. meeting_fr.json beside meeting.json) are separate canonical transcripts with separate mirrored speaker-map sidecars. Import may copy the base speaker map into the variant sidecar; see docs/runtime/host-stt.md (Multi-language variants).
- A file at transcripts_dir/*.json alone is not library-valid; library admission requires a managed artifact set (canonical JSON + valid import sidecar, with archived original path linkage).
- Host transcription helpers (whispermlx-missing, inbox-watch) must write raw engine output under transcripts/originals/ only. They refuse the managed library root (the directory that already contains metadata/ / imports/). They skip a stem when matching JSON already exists in originals/ or in the library root; they still never write into the library root. Admit via Import Transcript, Settings → Watcher, or optional inbox-watch --admit (subprocesses python -m transcriptx.admit_originals → admit_and_register; default off).
- Naming leaves room for future subtypes (diarised/, normalized/, export/).
data_dir: App-owned, persistent but partially reconstructable, not user-authored.
- groups/ and speaker_profiles/ are durable local project state (not safe to auto-delete). Speaker profile .cache/ is disposable; profiles/links/events/operations and profiles/assets/ (optional avatar WebP photos — face PII) are canonical (see docs/contracts/speaker_profiles_v1.md). Canonical voice evidence under speaker_profiles/voice/ is biometric-derived local state (see docs/contracts/speaker_profiles_voice_v1.md); .cache/voice/ and .cache/identify/ (fusion review dumps) are disposable. Ordinary exports must exclude voice/ and .cache/voice/. Auto-identify knobs: auto-identify.md.
- Speaker profiles contain real display names (PII) and may contain avatar photos and voice embeddings. Do not commit them. The in-repo default data/speaker_profiles/ is gitignored. For real use, point TRANSCRIPTX_SPEAKER_PROFILES_DIR (or TRANSCRIPTX_DATA_DIR) at a directory outside the git clone, the same way TRANSCRIPTX_TRANSCRIPTS_DIR / TRANSCRIPTX_OUTPUT_DIR keep metadata and outputs mountable off the repo root.
- Other subtrees (outputs, preprocessing, cache) remain reconstructable by re-running.
config_dir: User/app config, persistent, not safe to auto-delete.
- profiles/ lives under config_dir (user-editable config presets).
- config.json holds project settings including the Custom Questions library (analysis.llm_custom_qa.saved_questions).
- identify.json holds Settings → Speakers ingest defaults (auto_name / auto_link / style_only_apply; all default false). Not voice consent.
- With Docker Compose, set HOST_CONFIG_DIR to a host directory outside the git clone (same pattern as HOST_TRANSCRIPTS_DIR / HOST_OUTPUT_DIR) so Settings survive wiping ./data. Default remains ./data/.transcriptx.
outputs_dir: App-managed analysis outputs, reconstructable by re-running.
state_dir: App state (processing state, run/analysis locks, rename journal), persistent, reconstructable in part. Lives under data_dir/state/.
wav_backup_dir: Archive, persistent, not auto-clean unless explicit user action.

Canonical transcript validation¶

TranscriptX treats canonical transcript JSON as a typed, validated artifact, not just “any JSON file under transcripts_dir”.

All library-valid transcripts must satisfy the canonical schema in this section (schema_version / source / metadata / segments invariants). Operational validate/import: docs/runtime/host-stt.md (Python API).
Any API that accepts a “transcript path” must either:
- receive a pre-validated canonical transcript handle produced by the managed import workflow or a dedicated loader, or
- perform canonical validation itself (for example via validate_transcript_document) and fail closed on invalid or ambiguous data.
Callers and modules must not “guess” based on filenames or directory placement (for example, “this looks like a transcript, let’s try to use it”); loading arbitrary JSON from disk and treating it as canonical transcript data is a storage contract violation.

Where possible, public helpers should:

Take strongly-typed/canonical transcript descriptors instead of raw Path objects, or
Enforce canonical validation on entry before accessing transcript content.

Ad hoc file access logic that bypasses canonical validation is considered a bug and may be rejected in code review.

Final intended directory model¶

recordings_dir/                 # user library (mountable)
transcripts_dir/                # user library (mountable)
  imports/                      # ephemeral staging
  originals/                    # archived source files
  metadata/                     # managed import sidecars
  readable/                     # derived transcripts

config_dir/                     # configuration
  profiles/                     # module/workflow/STT/UI-layout named presets (not speaker profiles)
  install_profile               # optional marker: core | full
  config.json                   # project settings bag
  identify.json                 # Settings → Speakers ingest defaults (auto_name / auto_link / style_only_apply; default all false)
  interface_menus.json          # Settings → Interface action strips (schema v1; see docs/contracts/interface-menus.md)

data_dir/                       # app-managed working state
  groups/                       # group definition manifests (*.group.json); local user data — not tracked
  speaker_profiles/             # longitudinal speaker profiles, links, events, ops (canonical PII); override with TRANSCRIPTX_SPEAKER_PROFILES_DIR; see docs/contracts/speaker_profiles_v1.md and speaker_profiles_voice_v1.md
    profiles/                   # *.speaker_profile.json
      assets/{profile_id}/      # optional avatar.webp (face PII; include in backups)
    links/                      # *.speaker_link.json
    events/                     # *.speaker_event.json (filename stem = event idempotency id)
    operations/                 # *.op.json + staging/backup while active
    voice/                      # enrolled trusted-voice evidence (biometric-derived; durable on ./data bind mount; ordinary export excludes this tree)
      samples/ embeddings/ vectors/
      privacy.voice_settings.json
      active_generation.json generations/
    .cache/                     # disposable listing/aggregate caches only (.cache/voice/ and .cache/identify/ disposable)
  outputs/
    groups/                     # group analysis run outputs (per group uuid / run id)
  preprocessing/
  cache/
    audio_playback/
    voice/
  state/                        # processing state, locks, rename journal
    processing_state.json
    run_locks/                  # per-run writer lock files
    analysis_locks/             # per-transcript / per-group analysis-in-progress claims
    speaker_profiles.lock       # project operation lock for speaker profile mutations
  watcher/                      # directory watcher (G2) job records + activity; optional
    jobs/                       # *.json per watched-file job
    activity.jsonl              # append-only activity log
  backups/
    wav/
    processing_state/

This model separates:

User libraries: recordings, transcripts (mountable, never auto-clean).
Configuration: config_dir and profiles.
Application state and cache: data_dir (outputs, preprocessing, cache, state).
Backups and reproducibility artefacts: under data_dir/backups/.

Full-workspace backup¶

Operators can pack and restore the authoritative workspace (transcripts, config, durable data) as a portable ZIP with role-root remapping. Normative rules: contracts/workspace-backup.md. Operator guide: backup_and_restore.md. Default archives live under data_dir/backups/workspace/.

Renames¶

Managed transcript files participate in a broader contract that includes metadata sidecars, indices, and archival audio linkage. Moving them directly on disk can silently corrupt that contract.

Managed transcript renames must go through the storage rename service (rename_managed_transcript / web RenameService).
Direct filesystem renames of canonical transcripts (for example, calling Path.rename() on files under transcripts_dir or its metadata subtrees) are not supported and may leave metadata, indices, or cached state pointing at stale paths.
Archival/original audio under transcripts/originals/ (and wav_backup_dir) is never renamed. Associations stay via metadata.
Recordings working copies under recordings_dir may be renamed when the service classifies them as renameable and the target is free; a working-copy target collision blocks the entire rename.
Import metadata sidecars use the mirrored layout (metadata/imports/<rel>.import_meta.json). Legacy flat metadata/<stem>.import_meta.json is migrated fail-closed during rename (never left as the authoritative location after success).
A durable rename journal under state_dir/rename_journal/ records the full transaction plan at prepared, then post-commit progress (transaction_committed → finalized → reconciled → complete). Incomplete ops are recoverable via repair_managed_rename(operation_id) / discover_incomplete_renames(). Prepared-phase recovery is limited to deterministic classification (all committed / none started / ambiguous); partial in-flight transaction states require manual inspection.
Journal records include schema_version, planned slug mapping, and structured error history. Malformed journals are reported separately from incomplete ones.
Global lock: state_dir/managed_rename.lock (held through plan, transaction, finalization, and journal updates).
Dry-run may acquire the lock transiently but must make no persistent domain-state changes.

Contributors should treat direct filesystem moves of canonical transcript or metadata files as bugs.

Metadata mirroring invariant¶

Metadata sidecars must be discoverable and stable. To prevent layout drift, all transcript metadata locations obey a strict mirroring rule:

All transcript metadata paths must be derived via shared helper functions and must mirror the transcript-relative path under their metadata-kind subtree. Flat or ad hoc metadata paths are not permitted.
New code must obtain metadata paths only through central helpers (for example, helpers in paths.py or a dedicated metadata-paths module), never by manual string concatenation.
Introducing flat files like metadata/*.json that do not mirror the transcript-relative path is explicitly disallowed and will be treated as a regression.

Example mapping:

transcripts/foo/bar.json → metadata/imports/foo/bar.import_meta.json (import metadata sidecar)

This invariant allows future tooling to reliably locate and manage all metadata for a transcript without scanning arbitrary layouts.

imports/ staging semantics¶

The imports/ subtree exists purely as an internal staging area for the managed import workflow:

imports/ is an internal staging area used by the managed import workflow and is not part of the canonical storage contract or discovery surface.
No production feature (discovery UI, search, indexing, or analysis) may depend on the long-term presence, exact layout, or filenames under imports/.
Golden paths for discovery and analysis operate only on managed, library-valid transcripts under the canonical storage contract (canonical JSON + sidecars + archived original), not on raw files in imports/.

imports/ may evolve or disappear without changing the canonical storage contract; treat it as an implementation detail of ingestion.

Implementation pitfalls (Do/Don’t)¶

To keep the storage contract hardened, follow these guidelines:

Don’t open arbitrary .json files under transcripts_dir (or any subdirectory) and assume they are valid transcripts.
Do use the managed import workflow or canonical loaders and/or call the canonical validator before use.
Don’t write metadata files directly under metadata/ with custom layouts or flat filenames.
Do use shared metadata path helpers that mirror the transcript-relative path under the appropriate metadata-kind subtree.
Don’t call Path.rename() (or equivalent raw filesystem operations) on canonical transcript or metadata files.
Do use the storage rename service for any move/rename of canonical transcript artifacts.
Don’t assume that renaming a transcript always renames audio.
Do expect archival originals to stay put; only linked recordings working copies may rename when planned, and conflict blocks the whole rename.
Don’t build discovery, indexing, or production features on top of imports/ contents.
Do target only canonical managed storage for discovery and analysis.

These rules are intentionally strict; violating them risks subtle corruption of the storage model.

End-to-end storage flow (overview)¶

The following diagram summarizes the intended flow from raw inputs to canonical storage, emphasizing validation, metadata mirroring, and rename behavior:

flowchart LR
  rawInput["Raw input files"] --> importsStaging["imports/ staging"]
  importsStaging --> importWorkflow["Managed import workflow"]
  importWorkflow --> validator["Canonical transcript validation"]
  validator -->|"valid"| canonicalStore["Canonical transcripts"]
  validator -->|"invalid"| rejection["Fail closed / reject"]
  canonicalStore --> metadataHelpers["Metadata helpers (mirrored paths)"]
  metadataHelpers --> metadataStore["Metadata subtrees"]
  canonicalStore --> renameService["Rename service"]
  renameService --> canonicalStore
  renameService -.-> audioStore["Archival audio (stable paths)"]

Canonical validation is mandatory between staging and canonical storage; metadata helpers maintain mirrored layouts; the rename service does not rename archival audio; and imports/ is confined to staging and never used for discovery.

Contract violations¶

This section describes storage contract violations, how they are detected, and the expected behavior.

Invalid states (examples):
- Canonical transcript JSON under transcripts_dir without the required managed sidecars or archival/original linkage.
- Metadata files under metadata/ that do not mirror transcript-relative paths or are not derived via shared helpers.
- Ad hoc flat metadata layouts (for example, metadata/*.json that are not part of a mirrored subtree).
- Direct filesystem renames or moves of canonical transcripts or metadata that bypass the storage rename service.
- Discovery or analysis codepaths that depend on imports/ layout or treat imports/ contents as canonical.
Detection:
- Library admission and validation helpers that enforce canonical schema and managed transcript requirements.
- Central metadata path helpers that reject non-mirrored layouts or unexpected roots.
- Storage-level diagnostics and tests that scan for invalid paths or layout drift.
Expected behavior:
- Violations are treated as fail-fast for production codepaths: reject invalid artifacts, log clear errors, and avoid silently accepting ambiguous layouts.
- Tooling and maintenance scripts may offer guided repair (migration, cleanup) but must not silently reinterpret invalid layouts as valid storage.