Notebook corpus¶

Activation gate¶

This contract is runtime-normative for the bulk-import generation. The gate below is satisfied (corpus index, ImportRun/plan orchestration, lock order, corpus doctor checks, and acceptance suite green).

Shipped together:

Durable corpus index (transcribe.corpus-index) with atomic writers
ImportRun / ImportPlan persistence and commit orchestration
Corpus lock + lock-order enforcement
Corpus doctor checks listed in corpus-integrity.md
Acceptance suite green per corpus-integrity.md

transcribe.project schema_version 1 remains fully conformant without a corpus index: implementations must not require a corpus index to load, OCR, analyse, or export existing notebooks. New writers must not break v1 projects that lack corpus registration. Absence of corpus-index.json in a workspace means bulk-import is not yet used there; legacy discovery of project.json children continues.

This document (with source-asset.md, import-run.md, corpus-integrity.md) owns identity, ownership, ordering, and workspace corpus authority. project-on-disk.md remains sole authority for per-notebook directory layout and per-notebook ingest journal/locks.

Purpose¶

Transcribe is a durable notebook corpus. OCR, analysis, thumbs, and archive search are derived processes. An imported page is an archival object: once committed, its identity, managed original, human edits, OCR history, and notebook membership survive renames, folder moves, OCR model changes, analysis modules, and UI changes.

Hierarchy¶

Corpus (workspace)
  └── Notebook (notebook_id ≡ project.id)
        └── Page (page_id)
              ├── SourceAsset (source_id)     # primary imported bytes; PDF may back N pages
              ├── Render(s) (render_id)      # derived display pixels
              └── OCR attempts / edits       # page-result
        └── Analysis artifacts               # derived under analysis/

Product language “Project → Notebook” maps to Corpus → Notebook here so it does not collide with wire format transcribe.project.

Identity¶

ID	Rules
`notebook_id`	Generated UUID hex; immutable; never reconstructed from folder name, filename, title, or path
`page_id`	Generated UUID hex; immutable; never reconstructed from paths
`source_id` / `render_id`	Generated UUID hex; immutable

Wire alias: In transcribe.project v1, the field is id. Domain name is notebook_id. Invariant wherever both appear: notebook_id == project.id. Renaming “Notebook 17” → “Paris 2019” changes metadata only.

Filesystem layout is non-contractual for identity. Directory names under the projects root are implementation locators only. Domain IDs and the corpus index are authoritative for “which notebook exists” and “where it is managed.”

Ownership¶

A page belongs to exactly one notebook.
A SourceAsset belongs to exactly one notebook and backs one or more pages of that notebook.
A single-image import creates one source and one page. A PDF import creates one source and N pages (N = represented within-source page set).
OCR attempts and analysis artifacts are owned by the notebook (via page_id / project-local analysis/).
Loose arrays of unrelated page records outside a notebook entity are non-conformant.

Ordering (chosen representation)¶

Scope	Authoritative representation
Pages within a notebook	`project.pages` list order in `project.json`. Position = list index. Reorder permutes the list only; never changes `page_id`.
Notebooks within the corpus	`entries` list order in the corpus index. Position = list index. Reorder permutes the list only; never changes `notebook_id`.

Do not use parallel explicit ordinal integers as a second authority. Derived displays may show 1-based positions; they must recompute from list order.

page_index on a page is within-source identity/order (image → 0; PDF → PDF page index), not the notebook’s global page order. Global notebook order is solely project.pages list order.

Corpus index wire contract¶

Durable workspace document that locates managed notebooks without treating folder names as identity.

Field	Value
`format`	`transcribe.corpus-index`
`schema_version`	`1`
Location	`{TRANSCRIBE_DATA_DIR}/corpus/corpus-index.json` (create parent dirs on first write)
Writes	Atomic replace only (temp → fsync → `os.replace` → directory fsync), under the corpus lock

Document shape¶

{
  "format": "transcribe.corpus-index",
  "schema_version": 1,
  "updated_at": "<ISO-8601>",
  "entries": [
    {
      "notebook_id": "<uuid-hex>",
      "managed_relpath": "<path relative to TRANSCRIBE_PROJECTS_DIR>",
      "registered_at": "<ISO-8601>",
      "updated_at": "<ISO-8601>"
    }
  ]
}

Entry rules¶

entries list order is the authoritative notebook order.
notebook_id must equal project.id inside the managed notebook’s project.json.
managed_relpath is a mutable locator (may change if the directory is moved within the projects root). It must stay path-contained under TRANSCRIBE_PROJECTS_DIR and must resolve to a directory containing a valid project.json.
Duplicate notebook_id values are invalid.
Duplicate managed_relpath values are invalid.
Absence of the corpus index file means bulk import has not been used in that workspace yet; legacy discovery of project.json children continues. When the index is present, it is the durable locator set.

What the index is not¶

Not a search/timeline cache (that remains disposable archive SQLite).
Not a substitute for notebook entity records.
Not authoritative for page order, OCR, or analysis.

Workspace locking and lock order¶

Lock	Path / scope
Corpus lock	`{TRANSCRIBE_DATA_DIR}/corpus/.corpus.lock` — corpus index, ImportRun registry mutations, notebook registration
Notebook mutation lock	`<notebook>/.transcribe.lock` — `project.json`, sources/renders promote, page results, analysis RMW (existing)

Lock order (mandatory): always acquire corpus lock → notebook mutation lock. Never acquire in reverse. Never hold two notebook mutation locks while taking the corpus lock in between in a way that inverts order.

Serialization: commits that target the same notebook_id must be serialized (one active ingest transaction per notebook; see import-run.md and the single .ingest-journal.json rule in project-on-disk.md). Distinct notebooks may commit concurrently only if each holds its own mutation lock and the corpus lock is not held across long per-notebook work—corpus lock critical sections must stay short (index/run-registry RMW only).

Human metadata protection¶

Generalize the date_approved pattern:

Machine processes (import, OCR, re-analysis) may populate unapproved suggestions.
Approved human corrections must not be silently overwritten.
v1 normative field set: page diary date / date_approved / date_source (existing). Title/tags/future fields should reuse the same approved-vs-machine shape when added; do not invent one-off silent overwrite rules.

Authoritative vs derived¶

Authoritative	Derived / disposable
`project.json` entity + ordered pages	Archive SQLite / FTS
Managed source bytes + fingerprints	Thumbnails, `.cache/**`
Page results (OCR history, edits)	Library summary projections
Corpus index entries + order	Directory names as UX labels
ImportRun / ImportPlan records
Human-approved metadata

Indexes and caches may be rebuilt; IDs, provenance, approved metadata, OCR history, and ImportRun history must never be reconstructed by guessing.

Non-goals¶

Nested multi-notebook containers inside one project.json
Treating external scan folders as the durable unit
Reconstructing notebook_id / page_id from paths
Deserializing full OCR/analysis payloads merely to show the Library