Analysis module porting (from TranscriptX)¶

Planning map for which TranscriptX analysis modules to bring into Transcribe, how to adapt them for page/notebook text, and which to leave behind.

Core module set (Port early) is shipped — see [ROADMAP.md and [analysis_wave1_plan.md. Deferred reinterpretations and ocr_quality are not scheduled; current product focus is the usability wave ([ROADMAP.md Now · [usability_wave_plan.md). This map remains the disposition authority for deferred / later / out-of-scope rows when reopened.

Transcribe is page-first OCR text, not timed speaker segments. Modules that assume speakers, turns, audio, prosody, or ASR word confidence do not transfer as-is. See also [INTEGRATION_SEAM.md.

Disposition legend

Disposition	Meaning
Port early	Strong fit; port almost verbatim onto canonical analysis input (core: done)
Reinterpret	Useful idea; redesign semantics for notebooks
Later	Interesting after core analysis exists; may need a new module identity
Do not port	Intrinsically transcript/audio/interpersonal; out of scope
New (special case)	Notebook analogue of a TranscriptX idea; implement fresh, do not port the TX code

Slice column uses internal delivery ids from [analysis_wave1_plan.md. The core set was delivered as slices 1a–1e. Product sequencing: [ROADMAP.md.

Architecture (chosen)¶

Port analytical cores almost verbatim. Thin notebook adapters own project I/O. Modules must not be rewritten to understand notebooks, Page objects, or Streamlit state.

Contracts first: [analysis-document · [analysis-result · [analysis-run-storage · [notebook-eligibility · layout [project-on-disk. Pins: [dev/analysis_port_pins.md.

Managed notebook project (ingest copies sources/; external originals untouched)
     ↓
notebook_analysis_adapter (+ notebook_eligibility_v1 when required)
     ↓
AnalysisDocument (contract schema v1)
     ↓
ported TranscriptX module   (exact TX commit/file pin + semantic_class)
     ↓
analysis-result → project-local analysis/ storage (bound to project_id) → Notebook UI

Transcribe owns	Ported core owns
Managed project identity, page IDs, persistence under `analysis/`, invalidation, locking, UI	Scoring / ranking / clustering / inference on text+units

Copy modules with exact TX pins and parity / adaptation / fork classification. Resist extracting a shared transcriptx-analysis library until identical cores are obvious.
Core eligibility: sole named policy [notebook_eligibility_v1 — no ad-hoc per-module insight_eligibility stubs.
Small compatibility test corpus for TX ↔ Transcribe diffs (implementation-time).
Chronology = unit order + optional date — no synthetic wall-clock or fake speakers.

Full detail: [analysis_wave1_plan.md.

Notebook UI surfaces ↔ core modules¶

Shipped View pages are marked UI. Patterns remain payload feeds without a dedicated tab. People & places is a View page (People / Places with This notebook | All notebooks scope): read-models over published ner (optional Nominatim geocode + local cache). Moments is a Mood section; Ask notebook is a Summaries section.

Surface	Status	Core feeds
Overview	UI	`stats`, `lexical_diversity`, `ner`, `keyphrases`, `topic_modeling`, `wordclouds`, `understandability`
Themes	UI	`keyphrases`, `topic_modeling`, `bertopic`, `topic_shift`, `semantic_similarity`
People & places	UI	`ner` (map/list); `entity_sentiment` remains available as payload polish

Porting table¶

Module	TX UI group	Disposition	Slice	Notebook notes
`stats`	Foundations	Port early	1.1	Page/notebook length, token counts, distributions over units
`lexical_diversity`	Language & Meaning	Port early	1.1	Diversity metrics over notebook vocabulary
`understandability`	Language & Meaning	Port early	1.1	Readability / complexity of transcribed text
`wordclouds`	Visualisations	Port early	1.2	Baseline token cloud from `AnalysisDocument.text` (`enrichment_mode: baseline`); keyphrase enrichment deferred to deliberate later mode/`module_version` transition
`ner`	Language & Meaning	Port early	1.3	Entities across pages; evidence via `source_ref`; spaCy optional
`sentiment`	Language & Meaning	Port early	1.3	Unit-level polarity; chronology via order/date
`epistemic_markers`	Language & Meaning	Port early	1.3	Hedging / certainty markers in handwritten prose
`entity_sentiment`	Language & Meaning	Port early	1.4	Needs `ner` + `sentiment`
`keyphrases`	Language & Meaning	Port early	1.4	Use [`notebook_eligibility_v1`; do not pull TX `insight_eligibility`
`topic_modeling`	Language & Meaning	Port early	1c	Topics over page corpus; [`notebook_eligibility_v1`
`bertopic`	Language & Meaning	Port early	1c	Optional BERTopic extra
`semantic_similarity`	Language & Meaning	Port early	1c	Across pages; no multi-speaker gate
`topic_shift`	Dynamics & Flow	Port early	1c	Shifts along page order / dates, not timestamps
`emotion`	Language & Meaning	Port early	1d	Emotion labels on notebook text
`contextual_emotion`	Language & Meaning	Port early	1d	Context = neighbouring units by order
`fine_grained_emotion`	Language & Meaning	Port early	1d	Finer emotion taxonomy
`affect_tension`	Dynamics & Flow	Port early	1d	Needs `emotion` + `sentiment`
`moments`	Dynamics & Flow	Port early	1d	Notebook salience (no TX `momentum`/pauses)
`highlights`	Summary & Synthesis	Port early	1e	Quote-forward spans; [`notebook_eligibility_v1`
`summary`	Summary & Synthesis	Port early	1e	From highlights
`insights`	Summary & Synthesis	Port early	1e	Needs highlights + topics; `notebook_eligibility_v1`
`llm_summary`	Summary & Synthesis	Port early	1e	Optional local Ollama; honesty label
`llm_action_items`	Summary & Synthesis	Port early	1e	Tasks / decisions / open questions
`llm_custom_qa`	Summary & Synthesis	Port early	1e	Grounded QA with unit evidence
`narrative_summary`	Summary & Synthesis	Port early	1e	LLM narrative from deterministic summary
`politeness`	Speakers & Interaction	Reinterpret	2 (deferred)	→ tone / formality of notes (not interpersonal politeness)
`echoes`	Speakers & Interaction	Reinterpret	2 (deferred)	→ repeated ideas/phrases across pages or notebooks
`temporal_dynamics`	Foundations	Reinterpret	2 (deferred)	→ change through notebook chronology / page order
`momentum`	Dynamics & Flow	Reinterpret	2 (deferred)	→ density / idea-flow rather than conversational flow
`transcript_output`	Foundations	Reinterpret	2 (deferred)	→ clean notebook text / export surface
`simplified_transcript`	Foundations	Reinterpret	2 (deferred)	→ simplified / cleaned notebook text
`chart_descriptions`	Summary & Synthesis	Reinterpret	2 (deferred)	Still applicable once notebook analysis charts exist
`ocr_quality`	(new)	New (special case)	2 (deferred)	Do not port `transcript_quality`. Deferred: prefer second-pass LLM OCR cleanup/verification over a dedicated quality module. Revisit only if a clear analysis-owned gap remains.
`tics`	Foundations	Later	3	→ recurring phrases / habitual wording in notes
`insight_eligibility`	Foundations	Later	3	Survive if made content-generic (not transcript-genre gated)
`qa_analysis`	Speakers & Interaction	Later	3	Self-posed questions and subsequent answers in notes
`acts`	Speakers & Interaction	Later	3	→ note-type classification (observation / question / task / reflection)
`conversation_loops`	Speakers & Interaction	Later	3	Recurring unresolved themes — prefer a separate module rather than pretending it is the same
`interactions`	Speakers & Interaction	Do not port	4	Speaker turn-taking / equity; no speakers
`pauses`	Foundations	Do not port	4	Timed silence; no audio timeline
`transcript_quality`	Foundations	Do not port	4	ASR confidence scorecard; notebook `ocr_quality` analogue remains deferred (prefer OCR cleanup/verification)
`llm_speaker_summary`	Summary & Synthesis	Do not port	4	Speaker-conditioned LLM summary
`contagion`	Speakers & Interaction	Do not port	4	Interpersonal affect contagion unless deliberately redefined later
`voice_features`	Voice & Audio	Do not port	4	Audio feature extraction
`voice_mismatch`	Voice & Audio	Do not port	4	Voice / speaker-map mismatch
`voice_tension`	Voice & Audio	Do not port	4	Voice tension overlays
`voice_fingerprint`	Voice & Audio	Do not port	4	Speaker voice fingerprinting
`voice_charts_core`	Voice & Audio	Do not port	4	Voice chart gallery
`voice_contours`	Voice & Audio	Do not port	4	Pitch/prosody contours
`prosody_dashboard`	Voice & Audio	Do not port	4	Prosody / pitch family dashboard

Summary counts¶

Disposition	Count	Status
Port early (core / 1a–1e)	25	shipped
Reinterpret (deferred)	7	deferred (need unproven; see ROADMAP)
New special case (deferred)	1 (`ocr_quality`)	deferred (prefer OCR cleanup/verification)
Later	5	planned (after deepen-in-place)
Do not port	12	out of scope

Principles¶

Canonical units, not Page objects — adapters produce AnalysisDocument; cores stay TX-shaped.
Contracts first — schemas, outcomes/attempts, storage, eligibility, and dependency compatibility are CONTRACT-owned; PRODUCT summarises.
Exact pins + semantic class — no module lands without a [pin registry row (parity / adaptation / fork).
Managed-project storage — durable analysis under project analysis/; no global analysis authority; no in-place .transcribe/ layout.
Prefer deepen-in-place after a module lands; do not invent parallel IDs for the same user-facing object.
Reinterpretations keep the TX name only when semantics stay close; otherwise introduce a notebook-native id (e.g. ocr_quality, and a new id if conversation-loop analogues are rebuilt).
Do not port voice, prosody, pitch, pauses, interactions, speaker LLM summary, or interpersonal contagion as currently defined.
No TranscriptX runtime dependency — copy selected modules with exact pins; no imports from the TX package.
Provenance + compatibility corpus — every port records TX files + external analytical deps; fixtures support TX ↔ Transcribe diffs.
Copy first; shared library later — extract only when identical cores become obvious.