Source assets

Activation gate

Same gate as notebook-corpus.md — satisfied; this contract is runtime-normative. Existing SourceDocument fields in transcribe.project v1 remain the shipped source model; optional linkage fields are additive. This contract defines the durable semantics bulk import obeys and the integrity invariants notebook validation enforces (see Migration).

Purpose

Every imported image/PDF is a managed immutable original plus provenance. Runtime authority after commit is the managed copy and its content hash—not the external path that happened to be scanned.

Managed originals

  • Import copies source bytes into the notebook’s managed sources/ tree.

  • External path/filename are non-authoritative provenance only. They may move or disappear between scan and resume; idempotency must not key off them.

  • After commit, missing managed bytes (or fingerprint mismatch) is an integrity error, not a soft warning to re-read /Volumes/....

Identity vs fingerprint

Concept

Role

source_id

Immutable identity of the SourceAsset record

sha256

SHA-256 of the imported source bytes (JPEG/PNG/PDF as stored). Integrity + duplicate classification

External path/filename

Provenance display / audit only

Field semantics (aligned to current SourceDocument)

Field

Status

Semantics

source_id

Required

UUID hex

sha256

Required

Content hash of managed source bytes

media_type

Required

e.g. image/jpeg, image/png, application/pdf

original_filename

Required today

Basename/provenance label from import

stored_relpath

Required

Managed path relative to notebook root

page_count

Required

Exact size of the represented within-source page set (see below)

imported_at

Required

Import timestamp

render_dpi

Required

DPI used when renders were produced for this source

original_path

Optional / future

Full external path provenance; absence is valid

source_size_bytes

Optional / future

Byte length at import; absence is valid

import_run_id

Optional / nullable

Set when created by an ImportRun; must be null/absent for legacy sources predating bulk-import generation

bytes_present

Computed integrity state, not persisted authoritative metadata. Doctor/deep validation reports whether the managed file exists and matches sha256. Do not store a stale bytes_present: true flag on the entity.

PDF and page linkage

  • Do not put pdf_page_index on the SourceAsset. One PDF SourceAsset owns N pages.

  • Each page carries source_id + within-source page_index.

  • pdf_page_index lives on render provenance (and may be mirrored in staging journals). Where a render was produced from a PDF page, invariant: render.pdf_page_index == page.page_index for that page’s active render lineage when the source media_type is application/pdf.

  • For single-image sources, page_index is 0 and pdf_page_index is null/absent on renders.

page_count and within-source indices

For each source_id:

  1. Let P be the set of pages with that source_id.

  2. page_count must equal |P|.

  3. The set of page_index values in P must be exactly {0, 1, …, page_count - 1} (contiguous, starting at 0, no duplicates, no gaps).

  4. (source_id, page_index) is unique across the notebook.

Current shipped validation enforces contiguous unique indices via validate_project (see corpus-integrity.md notebook invariants).

Page ↔ render integrity invariants

Within a notebook (normative; doctor/validate_project must enforce):

  1. Unique (source_id, page_index) among pages.

  2. Active render belongs to the page’s source: the active render’s provenance must reference the same source_id as the page (via render bookkeeping already stored: source hash linkage and path layout pages/<source_id>/<page_index>/…). Explicitly: page.active_render_id resolves; that render’s source_sha256 equals the page’s SourceAsset sha256; render path containment uses the page’s source_id and page_index.

  3. render.source_sha256 == SourceAsset.sha256 for every render retained for pages of that source.

  4. PDF index coherence: for PDF sources, every page’s page_index equals the corresponding render’s pdf_page_index when present; page_count matches PDF page set represented.

  5. Dimension coherence: page.width/page.height equal the active render’s width/height.

  6. No unreferenced authoritative renders/sources unless explicitly permitted as a documented soft state. Default: every sources[] entry is referenced by ≥1 page; every renders map entry is the active render of exactly one page or is retained under an explicit future multi-render policy. Until multi-render history is a product feature, unreferenced renders are doctor errors (or warnings only if a migration note marks them transitional—default error for bulk-import generation).

Visual declutter provenance (additive)

Import may run visual declutter on the staged page PNG before it becomes the active render (workspace ingest.visual_declutter_enabled, default on). This is not OCR preprocess.

When present on a render, declutter fields are authoritative for that render’s pixels:

Field group

Role

declutter_state

One of disabled, enabled_noop, enabled_cropped, error_fallback

declutter_version / declutter_ops / declutter_params / declutter_identity_sha256

Frozen declutter identity (enabled flag + version + ordered ops + detection params)

Crop geometry

declutter_original_*, declutter_crop_*, declutter_inset_*

declutter_note

Bounded diagnostic (required non-empty for noop/error)

Absence of these fields means a pre-declutter-era render. Changing declutter config/version alone does not rewrite existing notebooks. New imports use the current identity. Explicit re-apply (Settings → Configuration → Re-apply visual declutter, Review → Cleanup, or ProjectService.reapply_visual_declutter) re-runs declutter on active renders (optionally a page subset): new render_id when pixels change; provenance-only update when bytes are unchanged. Re-apply does not restore already-cropped margins when enabled is false (records disabled provenance on current pixels only). Refuses while an OCR job lock is held. Does not re-run OCR. Ingest journals freeze declutter_identity_sha256 with page sha/geometry; crash recovery must not promote a journal whose identity no longer matches the effective setting.

Duplicate taxonomy

Classify before applying policy—never silently merge:

Class

Meaning

same_bytes_same_notebook

SHA-256 already present as a SourceAsset in the target notebook

same_bytes_other_notebook

SHA-256 present in a different notebook’s sources

same_filename_different_bytes

Provenance name collides; hashes differ

modified_replacement

Prior path/fingerprint pair no longer matches bytes (detected; not auto-applied)

pdf_page_duplicate_extract

Plan would create a second page for an already-represented (source, page_index)

Duplicate policies (first bulk-import release)

Allowed policies (named on the ImportPlan / import_policy_id):

Policy

Behaviour

skip_existing

See precise definition below

create_duplicate

Always create new source_id / page_id / render_id even when hashes match

replace_source is out of policy for the first bulk-import release. It must not be selectable; implementations must not offer silent replace.

skip_existing (precise)

  • May skip only when the candidate source SHA-256 already exists in the target notebook specified by the plan item (same_bytes_same_notebook).

  • Must not silently skip because the same bytes exist in another notebook.

  • Corpus-wide deduplication is allowed only when the plan explicitly sets a policy/flag requesting it (separate from default skip_existing). Default plans without that flag treat same_bytes_other_notebook as a distinct classification: either create_duplicate per policy or a validation finding—never an implicit skip.

  • Skips are recorded on the ImportRun item outcome with the existing source_id / page IDs that caused the skip.

Idempotency drivers

Idempotency keys for commit/resume are:

  1. Immutable planned IDs (plan_id, item_id, preallocated notebook_id / source_id / page_id / render_id)

  2. Source content sha256

  3. import_policy_id + plan fingerprint equality (import-run.md)

External paths and filenames must not drive idempotency.

Migration rules (before writing new linkage fields)

Before any writer persists import_run_id, original_path, or source_size_bytes into transcribe.project:

Decision (locked)

Rule

Additive optional fields on schema_version 1

import_run_id, original_path, source_size_bytes may be added as optional keys. Readers must treat absence/null as legacy-conformant.

No silent requiredness

Bulk-import writers may set them; OCR/review paths must not require them.

Schema bump reserved

Making any of these required, renaming id→notebook_id on the wire, or changing page_count semantics incompatibly requires schema_version bump and an explicit migration note—not drive-by implementation.

Until migration writers ship, doctors must accept legacy sources with only today’s required fields.

Non-goals

  • Content-defined merging of near-duplicate images

  • Deduplicating bytes into a global content-addressed blob store in v1 (notebook-local managed copies remain the unit)

  • Using filename lexicography as identity