Theme A — Insights & analysis quality¶

Status: active (1.x)
Last updated: 2026-08-10

Roadmap home: docs/ROADMAP.md §A
Product constraint: Prefer deepen-in-place; label Deterministic vs Local AI honestly; do not keep weak deterministic fallbacks merely to claim non-AI coverage (PRODUCT.md).

Goal¶

Stronger deterministic / hybrid insights: clearer, more useful non-LLM outputs with less noise; trustworthy empty/partial states; hybrid LLM paths inherit a cleaner deterministic base.

Locked decisions¶

Shared signal first — raise quality in phrase_quality + insight_eligibility before inventing new insight detectors.
Insights themes use theme_label_policy — same bar as summary theme labels (stricter than content_phrase_policy).
Abstain over pad — if fewer than min_themes_for_signal themes pass the floor, emit empty themes + status: insufficient_signal rather than weak fillers.
Schema v3 — each theme/idea row may carry evidence_quote_ids, confidence, preference_tier, and score breakdown; bump schema_version to 3.
Topic soft boost only — topic_modeling may boost an existing theme when labels overlap; it never invents themes. Dependency remains soft (missing topics = no boost).
Overview focus honesty — deterministic overview omits a focus clause when no high-tier theme labels survive; no fallback to emblematic filler phrases for focus text.
Hybrid inherits base — narrative_summary continues to rewrite only the deterministic summary findings; improving summary improves hybrid without prompt inventiveness.
B18 LLM insight narratives — deferred until provenance fields (P2-lite on schema v3 rows) are stable; Local AI only, never replacing deterministic theme lists.

Signal rules¶

Layer	Rule
Phrase hard rejects	Empty, all-stopwords, discourse formulas, light-verb constructions, tic/discourse mask, pronoun shards, no content token, short shards
Eligibility floor	Default `min_score=0.18`; single-token candidates require `spread > 0` or `recurrence > 0`
Insights compose	Diversity via `select_diverse_themes`; caps from `analysis.insights` config
Confidence bands	`high` if total≥0.55 and (spread≥0.15 or recurrence≥0.2); `medium` if total≥0.35; else `low` (low rows may be dropped by floor)

Schema v3 sketch¶

{
  "schema_version": 3,
  "status": "ok | insufficient_signal",
  "status_reason": null,
  "key_themes": [
    {
      "phrase": "budget risk",
      "score": {"total": 0.72, "spread": 0.3, "recurrence": 0.4},
      "confidence": "high",
      "preference_tier": 0,
      "evidence_quote_ids": ["q:..."],
      "topic_corroborated": false
    }
  ],
  "recurring_ideas": [],
  "style_markers": {},
  "notable_moments": [],
  "phrase_quality_version": 3
}

Empty-state matrix (UI)¶

Condition	User-facing copy
Insights module not run / failed	Existing quiet unavailable / module-required hint
`status == insufficient_signal` (content focus)	“Not enough clear content themes in this transcript.”
Eligibility / highlights missing for content block	“Content themes need Insight eligibility and Highlights in this run.”
Style markers empty	“No style indicators.” (unchanged)
Deterministic summary abstains on focus	Overview still shows speaker count / duration; no invented focus clause

Config knobs (`analysis.insights`)¶

Field	Default	Role
`top_themes`	8	Cap for key themes
`top_recurring_ideas`	8	Cap for recurring ideas
`top_notable_moments`	8	Cap for notable moments
`min_theme_score`	0.18	Compose-time floor (aligned with eligibility)
`min_themes_for_signal`	2	Below this → abstain
`overview_theme_cap`	5	Suggested UI cap on Overview
`topic_boost`	0.05	Soft score boost when topic label overlaps

Eligibility: analysis.insight_eligibility.min_score default 0.18; require_spread_or_recurrence_for_singletons default true.

Acceptance checklist¶

Banned fillers never appear in insights.key_themes or summary.key_themes (unit + smoke)
Abstention path covered when only weak phrases exist
Evidence quote ids present when highlights themes attach
Deterministic overview omits filler focus
Commitments require content beyond light-verb stems
Insights UI hides empty/abstention sections instead of sparse tables
PHRASE_QUALITY_VERSION bumped when hard-reject semantics change
Corpus before/after notes for 2–3 English meetings (manual)

Out of scope (this theme wave)¶

New module IDs
Multilingual phrase-resource expansion (Theme M / known limitation)
SQLite analytics (Theme J)
Reintroducing Insights → Analysis tab
Full B18 Local-AI insight narratives (follow-on after P2-lite)

Corpus probe notes (deep-test 2026-08-10)¶

Run	Transcript	Insights status	Notable themes
mini Python	`tests/fixtures/mini_transcript.json`	`ok`	google cloud support, new york stakeholders, market street
large Theme A modules	`tests/fixtures/analysis_probes/large_norm.json` (1007 segs / 13 speakers)	`ok` after log-freq + greeting rejects (was `insufficient_signal` with linear freq + min_score 0.28)	kitchen fairy, cape town, south africa, social media
group	mini + large	member runs green; group finalize `partial` with manifest/run_results present	—

Hardening during deep-test: log1p frequency scoring; greetings/et cetera discourse rejects; eligibility/insights floor 0.18; emblematic-phrase fallback before abstention; topic_modeling optional so insights stays on quick/balanced.