Analysis run storage

Durable analysis artifacts live inside the managed notebook project. Top-level paths are owned solely by project-on-disk.md. Result envelope semantics: analysis-result.md. Input fingerprint: analysis-document.md. Eligibility contribution: notebook-eligibility.md.

Ownership

• Transcribe persistence owns writes; analysis cores are pure over canonical I/O

• Authoritative outputs are project-local under analysis/ — never a global per-module analysis tree

• Workspace archive.sqlite (and any discovery index) may point at projects but is never analysis authority

Layout (reference only)

Per project-on-disk.md:

Path	Role
analysis/	Durable analysis artifacts; optional until first write
.cache/analysis/	Disposable acceleration only

Exact filenames, staging directories, and payload file formats under analysis/ are implementation-defined but must obey the publish, identity, and atomicity rules below. Illustrative pattern:

analysis/
  runs/<run_id>.json        # batch AnalysisRunPlan + progress (optional)
  <module_id>/
    published.json          # current published reusable artifact (if any)
    attempts/<attempt_id>.json

Creating analysis/ on first write is not a project-layout migration; projects without it remain valid. The runs/ directory is reserved and is never treated as a module id.

Project identity binding

Every durable artifact and every cache lookup must include the canonical project_id from project.json, not merely the filesystem path.

Lookup key: (project_id, module_id, cache_identity)

• Moved projects remain valid when project_id is unchanged

• Cross-project reuse is impossible: path coincidence must not produce a cache hit

Atomicity and locks

Do not hold mutation_lock while running analysis cores. Long compute must not block unrelated project writes.

Required sequence:

1. Under mutation_lock (short): allocate attempt_id, persist attempt_state: running attempt artifact, record planned cache_identity / content fingerprint

2. Release lock; run the module unlocked

3. Persist the terminal attempt artifact atomically (immutable after leaving running)

4. Under mutation_lock (short): rebuild current document fingerprint / cache_identity; if stale vs the attempt’s planned identity → retain attempt, do not update published; else if outcome is cacheable → atomically replace published.json

5. Release lock

Attempt file writes and published-pointer swaps use write_json_atomic. Partial writes must not become published.

Layout

analysis/
  runs/<run_id>.json
  <module_id>/
    published.json
    attempts/<attempt_id>.json

Creating analysis/ on first write is not a layout migration. Skip the reserved runs/ name when scanning module directories.

Analysis batch lock

At most one analysis batch run per project across processes, held via .transcribe.analysis.lock (see project-on-disk.md). Long module compute holds this lock for the run lifetime and must not hold mutation_lock.

Frozen AnalysisRunPlan

Batch Analyse launches freeze an immutable AnalysisRunPlan before any module runs: ordered module ids, optional detector ids, optional question text, EffectiveConfig snapshot + config fingerprint, text-model identity when LLM modules or detectors are included, and preset identity (preset_key, preset_content_version, preset_policy_fingerprint). Workers consume the plan (bound config + frozen model identity), not live UI/settings. Mid-run settings / text-model / module-list changes apply to the next run only. Notebook content edits mid-run still use publish revalidation (stale_at_publish) — text is not frozen as execution authority.

plan_hash (preflight bind)

plan_hash is the hex SHA-256 of a canonical JSON object over execution-significant fields. Exclude ephemeral run_id, created_at, and plan_hash itself. Required body fields:

Field	Notes
project_id	from project.json
module_ids	ordered freeze list
detector_ids	ordered detectors (may be empty); run after modules via DetectionService
question_text	or null
effective_config	full EffectiveConfig snapshot
text_model	frozen model object or null
config_fingerprint	plan config fingerprint
preset_key	quick / balanced / thorough / custom (or null)
preset_content_version	integer content generation (Custom may use 0)
preset_policy_fingerprint	SHA-256 of policy body (or Custom module/detector-list fingerprint)

Preflight bind rule: the UI freezes the plan at launch confirm and stashes {plan, plan_hash}. Start must deserialize that plan and refuse when recomputed plan_hash ≠ stored hash. Start must not re-snapshot live settings/config. Coordinator start also refuses a tampered or empty plan_hash.

Durable run records (format: transcribe.analysis-run) live under analysis/runs/<run_id>.json. They are history/progress only and never replace module publish authority. Run records include plan_hash and preset identity fields alongside the embedded plan.

Reopen reconciliation

When the project is opened/loaded and the analysis lock is free: every attempt still running becomes interrupted, and every non-terminal run record becomes interrupted. Reconciliation must not clear or rewrite a valid published.json. Do not gate analysis reconcile on the OCR job lock. Process death does not auto-resume a batch; the user re-launches and published cache hits skip completed modules.

Cache-hit validation

Matching cache_identity is necessary but not sufficient. Refuse reuse when the published envelope fails schema validation, module_version disagrees with the requested module pin, the artifact is unparsable/corrupt, or required fields are missing.

Attempt history vs published result

Store	Role
Attempt history	Retained execution records (running / succeeded / failed / cancelled / interrupted)
Published result	Single reusable artifact per (project_id, module_id)

Publish rules (normative detail in analysis-result.md):

• Only attempt_state == succeeded with outcome ∈ {success, skipped_not_applicable, unavailable_dependency, insufficient_data} may update the published result

• running / failed / cancelled / interrupted must never replace the last published reusable artifact

• Reconciliation (running → interrupted when lock free) updates history only

A cache hit that skips rerun requires: published artifact present and schema-valid, project_id match, cache_identity match, module_version match, and outcome in the cacheable terminal set above.

Forbidden: cache keys of only (project_id, module_id), filesystem path, or raw text hash without the full cache_identity object.

Cache identity

cache_identity is the hex SHA-256 of a canonical JSON object (cache_identity_version: 1) with sorted keys and compact UTF-8 JSON (same serialization discipline as the analysis-document content fingerprint). Required fields:

Field	Source
cache_identity_version	1
project_id	project.json
module_id	module
content_fingerprint	analysis-document.md algorithm
content_fingerprint_version	1
module_version	Transcribe module version / code fingerprint
adapter_version	adapter
granularity_version	document
split_profile	document
config_fingerprint	relevant module configuration (canonical subset)
eligibility_policy_id	e.g. notebook_eligibility_v1 when used; else null
eligibility_policy_version	when used; else null
eligibility_fingerprint	hash of eligibility output when policy used; else null
parents	sorted array of {module_id, cache_identity, outcome} for required and optional parents actually consumed
lexicon_or_model	package/lexicon/model identities affecting reproducibility
resolved_model_digest	when applicable; else null
llm	when applicable (below); else null

Identity therefore varies with text/order/included pages and adapter granularity/splitting, module configuration, dependency outputs, algorithm/schema version, lexicon/model version or digest, and relevant prompt/runtime parameters — not content alone.

LLM llm object (when module uses a generative model)

Field	Required
prompt_or_template_version	yes
generation_settings	yes (canonical subset: temperature, max tokens, etc.)
grounding_strategy_id	yes
chunking_policy_id	yes
reduction_policy_id	yes (notebook_map_reduce_v1 for LLM modules)
token_estimator_id	yes (whitespace_tokens_v1 for LLM modules)
question_text	yes for llm_custom_qa; else null
resolved_model_digest	yes when model resolved; else null (preflight → unavailable_model)
input_fingerprint	yes (excerpts actually supplied / reduction fingerprint — not whole-document dump)

Frozen LLM policy ids (must match analysis-result.md):

Field	Allowed LLM policy values
chunking_policy_id	notebook_chunks_units_v1 (token budget; sub-split oversized units)
reduction_policy_id	notebook_map_reduce_v1
token_estimator_id	whitespace_tokens_v1
grounding_strategy_id	ground_doc_chunks_v1 | ground_highlights_summary_v1

Dependency compatibility (sole normative hard DAG)

PRODUCT docs may list human-readable relationships including soft enrichments. This contract owns hard-parent compatibility. Consumers must not silently reuse stale or differently configured parents.

Required (hard) parents (core set)

Consumer	Required parents	Acceptable parent outcomes
entity_sentiment	ner, sentiment	success
affect_tension	emotion, sentiment	success
summary	highlights	success
insights	highlights, topic_modeling	success
narrative_summary	summary	success

For each required parent, the consumer must verify:

1. Published parent exists for the same project_id

2. Parent outcome is in the acceptable set

3. Parent cache_identity equals the identity recorded in the consumer’s planned parents list (content/config/eligibility alignment)

On failure → commit reusable terminal with outcome: unavailable_dependency when attempt completes cleanly; never run the consumer core against mismatched parents.

Optional enrichments (non-blocking)

These parents enrich payloads when present and compatible; absence must not fail the consumer. When consumed, their {module_id, cache_identity, outcome} enter parents and thus cache identity. When absent, the consumer runs its documented baseline path (never an ad-hoc stub).

Optional-parent resolution must precede cache_identity construction and cache lookup. Only parents actually consumed enter parents. Baseline wordclouds mode locks wordclouds to enrichment_mode: "baseline": keyphrases is never consumed even when a compatible published success exists (absent / incompatible / failed / non-success / success → all ignored). Enrichment requires a deliberate later mode/module_version transition so enabling enrichment cannot silently change baseline identity or outputs.

Consumer	Optional parent	Baseline when absent
wordclouds	keyphrases	Token/frequency cloud from document text only (enrichment_mode: "baseline" in baseline wordclouds always takes this path)
topic_modeling	keyphrases	Model without keyphrase prior/seed enrichment
bertopic	keyphrases	BERTopic without keyphrase enrichment
contextual_emotion	neighbouring-unit window only	N/A (no parent module; window is config)
moments	emotion, sentiment, topic_shift (soft features)	Notebook salience features still computed; missing soft signals → lower feature set + warning, not hard fail
insights	—	Hard parents only; no TX insight_eligibility
llm_* / narrative_summary	deterministic summary / highlights as grounding context	narrative_summary still hard-deps summary; other LLM modules may ground on document text alone per module notes

Fallbacks (core set)

Situation	Behaviour
TX insight_eligibility	Not ported. Sole substitute: notebook_eligibility_v1 for the modules listed there
TX momentum (for moments)	Not ported. moments is fork / notebook salience — not a soft import of momentum
BERTopic extra missing	skipped_not_applicable / capability unavailable_extra — do not fall back to silently pretending LDA is BERTopic
Transformer emotion missing	lexicon/emotion path remains; heavy modules → unavailable_extra or unavailable_model
Ollama missing	LLM modules → unavailable_model; deterministic synthesis path unaffected

Modules not listed under Required (hard) parents have no hard parent dependencies under this contract version.

Non-goals

• Redefining top-level project layout (owned by project-on-disk)

• Global authoritative analysis storage

• Cache hits keyed only by filesystem path