Corpus integrity

Activation gate

Same gate as notebook-corpus.md — satisfied. Notebook-local and corpus-wide doctor checks below are required; the acceptance suite is the executable bar for supported bulk-import UI/CLI.

Doctor scopes

Scope

Responsibility

Notebook doctor

Extend today’s per-project DoctorService: structural + deep hash + source/render invariants in this contract

Corpus doctor

Workspace-wide: corpus index, ImportRuns, cross-notebook ID uniqueness, locator containment

Both may run offline. Deep mode rehashes managed source and render bytes.

Notebook invariants

In addition to existing validate_project / page-result checks, enforce:

  1. Unique page_id, source_id, render_id within the notebook

  2. Unique (source_id, page_index) among pages

  3. For each source: pages’ page_index set equals {0..page_count-1} and |pages| == page_count

  4. Every page.active_render_id resolves; active render belongs to the page’s source; render.source_sha256 == SourceAsset.sha256

  5. PDF coherence: when media_type is PDF, render pdf_page_index (when present) equals page.page_index

  6. page.width/height equal active render dimensions

  7. Managed source/render files exist (and match hashes in deep mode)

  8. No orphan page-result files without a page (warning or error per existing practice); no unreferenced authoritative source/render records (source-asset.md)

  9. Approved date metadata satisfies existing date invariants

  10. Well-formed ingest journal or quarantined corrupt journal — never silent delete (import-run.md)

Corpus invariants

  1. Every corpus index notebook_id loads a project.json whose id == notebook_id

  2. Every index managed_relpath is unique, contained under TRANSCRIBE_PROJECTS_DIR, and resolves

  3. notebook_id uniqueness across the corpus index

  4. Global uniqueness of page_id, source_id, and render_id across all registered notebooks (IDs are corpus-wide capable; collisions are errors even if folders differ)

  5. Index entry order is well-formed (list); no duplicate entries

  6. ImportRun files that claim committed IDs: those IDs resolve in the cited notebook, or the item is explicitly skipped/failed with recorded reason

  7. At most one live .ingest-journal.json interpretation per notebook; corrupt journals quarantined and reported

  8. Corpus lock / notebook lock order is an implementation duty; doctor does not prove lock order but may flag concurrent-journal anomalies

Repair and rebuild boundaries

May rebuild / delete-and-recreate

Must never guess-rebuild

Archive SQLite / FTS

notebook_id, page_id, source_id, render_id

Thumbnails / .cache/**

Human-approved metadata

Disposable Library summary projections

Provenance, OCR attempt history, edits

Optional ImportRun registry index file (from run files)

Managed source bytes

ImportRun immutable plan bodies and committed item outcomes

Corpus index entries (may re-register a known project.json only via explicit recovery that reads project.id, never invents IDs)

Supported recovery paths must leave doctor green (or only documented warnings) when completed successfully.

Migration enforcement

Before writers emit optional SourceAsset linkage fields or ImportRun IDs into transcribe.project:

  • Follow additive optional v1 rules in source-asset.md

  • Doctors must accept legacy sources without import_run_id / original_path / source_size_bytes

  • After writers ship, doctors may warn on committed ImportRun items whose sources lack import_run_id only when policy requires linkage for that generation—default: warn, do not invalidate legacy notebooks

Acceptance gate (executable)

Bulk-import implementation is supported (UI and CLI bulk paths) when the synthetic multi-notebook corpus suite passes all of the following — and that suite is green:

  1. Deterministic crash-injection at each boundary in import-run.md (corpus registration, notebook creation, source promotion, render promotion, project.json commit, ImportRun item commit, final run-state commit), with resume producing the authoritative winner and no duplicate committed IDs

  2. Retry / idempotency under exact plan_id + plan_fingerprint + import_policy_id equality: second commit is a no-op for committed items; failed items may progress without regenerating IDs

  3. Duplicate policy: skip_existing skips only same SHA in the target notebook; same bytes in another notebook are not silently skipped without explicit corpus-wide dedupe in the plan; create_duplicate always allocates the preallocated distinct IDs

  4. Corpus-index corruption / rebuild: corrupt index quarantined/reported; recovery re-registers from authoritative project.json IDs without inventing new notebook_ids; locator uniqueness restored

  5. Deep doctor green (or documented warnings only) after every supported recovery path on the synthetic corpus. Retained quarantine artifacts after a successful index rebuild are warnings (corpus_quarantine_present), not errors — operators may delete them after review.

  6. Fixture coverage includes: many notebooks × many pages, duplicate bytes, renamed external files, interrupted imports, missing managed sources, reordered pages, failed OCR, re-import, legacy schema v1 notebooks, Unicode/weird filenames, PDF split provenance, ordering ambiguity refused at validate, malformed journal quarantine. Cancel paths must distinguish cancelled vs cancelled_with_commits without rolling back committed items.

Acceptance bar: no page identity loss; no silent overwrite of approved human metadata; deterministic resume; doctor green (warnings-only allowed for documented quarantine retention) after every supported recovery path.

Non-goals

  • Automatic healing that invents missing IDs or approved metadata

  • Treating archive SQLite as an authority to repair notebooks from

  • Shipping bulk import behind a feature flag without this suite green (gate is closed; suite must remain green)